0 00:00:00,000 --> 00:00:30,000 Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/321 Thanks! 1 00:00:09,280 --> 00:00:10,900 Warm round of applause for our speakers. 2 00:00:18,920 --> 00:00:19,849 Thank you. 3 00:00:19,850 --> 00:00:21,349 My name is Dr. Drew. 4 00:00:21,350 --> 00:00:23,119 I'm a graduate student at the University 5 00:00:23,120 --> 00:00:25,099 of Michigan, where my research probably 6 00:00:25,100 --> 00:00:26,659 focuses on what can we learn about 7 00:00:26,660 --> 00:00:28,999 security from these large data 8 00:00:29,000 --> 00:00:31,159 analysis. Since last 9 00:00:31,160 --> 00:00:32,239 year, you've heard about some of my 10 00:00:32,240 --> 00:00:33,859 research from Alex Halderman, including 11 00:00:33,860 --> 00:00:35,569 work on ZENAB and how do we scan the 12 00:00:35,570 --> 00:00:36,739 entire Internet? 13 00:00:36,740 --> 00:00:38,809 Some work on we cryptographic 14 00:00:38,810 --> 00:00:40,979 keys, the ecosystem. 15 00:00:40,980 --> 00:00:43,579 I'm here today to talk about Heartbleed. 16 00:00:43,580 --> 00:00:45,409 What happened when Heartbleed actually 17 00:00:45,410 --> 00:00:46,369 hit the Internet? 18 00:00:46,370 --> 00:00:47,370 What was the aftermath 19 00:00:48,920 --> 00:00:50,299 in April 2014? 20 00:00:50,300 --> 00:00:52,399 Open SSL disclose a catastrophic bug in 21 00:00:52,400 --> 00:00:54,589 the implementation of the Tea Party 22 00:00:54,590 --> 00:00:56,749 extension, and the vulnerability 23 00:00:56,750 --> 00:00:58,249 essentially affected every piece of 24 00:00:58,250 --> 00:01:00,319 software that used open SSL to 25 00:01:00,320 --> 00:01:02,449 facilitate close connections. 26 00:01:02,450 --> 00:01:03,919 This included everything from Web 27 00:01:03,920 --> 00:01:05,749 servers. We heard a lot about Engine X 28 00:01:05,750 --> 00:01:07,849 and Apache, but also affected everything 29 00:01:07,850 --> 00:01:09,019 such as Tor. 30 00:01:09,020 --> 00:01:10,999 It affected the Bitcoin client, a lot of 31 00:01:11,000 --> 00:01:13,759 other pieces of software, 32 00:01:13,760 --> 00:01:15,889 and in the end it allowed attackers 33 00:01:15,890 --> 00:01:18,139 to essentially remotely steal any piece 34 00:01:18,140 --> 00:01:20,269 of information that potentially sat in 35 00:01:20,270 --> 00:01:22,279 memory of the SSL client. 36 00:01:22,280 --> 00:01:23,869 And this might include logins. 37 00:01:23,870 --> 00:01:26,029 It might include your credit card number 38 00:01:26,030 --> 00:01:28,009 that you had just transferred to Amazon, 39 00:01:28,010 --> 00:01:29,929 maybe included your cryptographic private 40 00:01:29,930 --> 00:01:32,209 key that you're using to facilitate 41 00:01:32,210 --> 00:01:33,349 connections. 42 00:01:33,350 --> 00:01:35,449 And it affected a large number of 43 00:01:35,450 --> 00:01:37,249 sites, not only just a large number of 44 00:01:37,250 --> 00:01:39,259 software products, but in the end, a huge 45 00:01:39,260 --> 00:01:40,609 percentage of the Internet was actually 46 00:01:40,610 --> 00:01:42,299 impacted in our estimates. 47 00:01:42,300 --> 00:01:43,759 Somewhere between twenty four to fifty 48 00:01:43,760 --> 00:01:46,489 five percent of all Web websites 49 00:01:46,490 --> 00:01:47,569 were initially vulnerable. 50 00:01:47,570 --> 00:01:49,549 Could have had this information stolen 51 00:01:49,550 --> 00:01:50,959 when the bug was first reported. 52 00:01:52,310 --> 00:01:54,889 So the T-West HAPI extension, 53 00:01:54,890 --> 00:01:56,629 I hadn't heard of it. I bet a lot of you 54 00:01:56,630 --> 00:01:58,249 hadn't heard of it prior to the 55 00:01:58,250 --> 00:02:00,199 Heartbleed exploit. 56 00:02:00,200 --> 00:02:02,359 The extension was released in 2012 57 00:02:02,360 --> 00:02:04,039 and essentially allows either end point 58 00:02:04,040 --> 00:02:06,229 of a connection to ask its partner, 59 00:02:06,230 --> 00:02:07,999 are you still there? 60 00:02:08,000 --> 00:02:10,339 And the partner essentially replies, Yes, 61 00:02:10,340 --> 00:02:11,989 I am still around, keep the connection 62 00:02:11,990 --> 00:02:13,909 open. And for the most part, we never use 63 00:02:13,910 --> 00:02:16,219 it heals heartbeat extension because 64 00:02:16,220 --> 00:02:18,349 we used Helus over TCP 65 00:02:18,350 --> 00:02:20,359 and TCV takes care of this on a lower 66 00:02:20,360 --> 00:02:21,559 network layer and make sure that our 67 00:02:21,560 --> 00:02:22,969 connection is still alive. 68 00:02:22,970 --> 00:02:24,979 But US is still designed to work with 69 00:02:24,980 --> 00:02:27,289 other lower level protocols such as UDP, 70 00:02:27,290 --> 00:02:28,459 where you might not have the session 71 00:02:28,460 --> 00:02:29,959 management and you just want to be able 72 00:02:29,960 --> 00:02:31,399 to say, are you still present without 73 00:02:31,400 --> 00:02:34,189 having to renegotiate an entire session 74 00:02:34,190 --> 00:02:35,479 in the protocol? It's actually very 75 00:02:35,480 --> 00:02:37,549 simple. Either end of the end point, 76 00:02:37,550 --> 00:02:38,869 either endpoint the connection can 77 00:02:38,870 --> 00:02:40,309 essentially say to his partner, are you 78 00:02:40,310 --> 00:02:41,959 there? And it does this by essentially 79 00:02:41,960 --> 00:02:44,119 sending some piece of data and some 80 00:02:44,120 --> 00:02:45,739 ran the padding and then asking its 81 00:02:45,740 --> 00:02:48,049 partner to repeat that data back. 82 00:02:48,050 --> 00:02:50,209 And if partner repeats the data back, it 83 00:02:50,210 --> 00:02:51,259 knows it's still there and it doesn't 84 00:02:51,260 --> 00:02:53,059 terminate the connection. 85 00:02:53,060 --> 00:02:54,679 So very simple. 86 00:02:54,680 --> 00:02:56,749 The bug existed in that both 87 00:02:56,750 --> 00:02:58,849 a length header was sent and 88 00:02:58,850 --> 00:03:01,009 then a set amount of data was set and 89 00:03:01,010 --> 00:03:01,999 an open SSL. 90 00:03:02,000 --> 00:03:04,309 We trusted the length header 91 00:03:04,310 --> 00:03:06,409 and essentially would echo back that 92 00:03:06,410 --> 00:03:08,479 amount of information and in the memory, 93 00:03:08,480 --> 00:03:10,159 regardless of how much data had actually 94 00:03:10,160 --> 00:03:11,359 been specified. 95 00:03:11,360 --> 00:03:13,399 So if I sent you for bytes of data but I 96 00:03:13,400 --> 00:03:15,229 told you I sent you eight bytes of data, 97 00:03:15,230 --> 00:03:17,089 you'd echo back eight bytes of data. 98 00:03:17,090 --> 00:03:18,769 And if those extra for bytes of data 99 00:03:18,770 --> 00:03:20,389 happened to be my username and password, 100 00:03:20,390 --> 00:03:22,189 you would happily echo those back to the 101 00:03:22,190 --> 00:03:23,190 client. 102 00:03:24,410 --> 00:03:26,239 So the fix is very simple. 103 00:03:26,240 --> 00:03:28,309 We check the length field and we check 104 00:03:28,310 --> 00:03:29,509 the amount of data. And if the length 105 00:03:29,510 --> 00:03:31,159 field is longer than the amount of data, 106 00:03:31,160 --> 00:03:33,169 then we throw away the request. 107 00:03:33,170 --> 00:03:34,909 But the same time, the effect is rather 108 00:03:34,910 --> 00:03:36,019 catastrophic. 109 00:03:36,020 --> 00:03:38,119 Any piece of information up to 110 00:03:38,120 --> 00:03:40,129 two to six bytes after 111 00:03:41,570 --> 00:03:43,189 that data could be echoed back. 112 00:03:43,190 --> 00:03:44,689 And depending on how your memory is laid 113 00:03:44,690 --> 00:03:45,799 out, this could be your login 114 00:03:45,800 --> 00:03:47,089 credentials. This could be a key. 115 00:03:47,090 --> 00:03:48,090 It could be anything else. 116 00:03:49,370 --> 00:03:51,409 And this affected a very large number of 117 00:03:51,410 --> 00:03:53,089 software. A lot of pieces of software 118 00:03:53,090 --> 00:03:54,499 actually end up using open SSL to 119 00:03:54,500 --> 00:03:56,839 facilitate connections. 120 00:03:56,840 --> 00:03:59,029 In the case of Ingenix, in case of 121 00:03:59,030 --> 00:04:01,069 Pache, this is used to serve as sheet 122 00:04:01,070 --> 00:04:03,439 sites in terms of the Bitcoin client. 123 00:04:03,440 --> 00:04:06,169 If you connected to a vault to 124 00:04:06,170 --> 00:04:08,479 a malicious server, 125 00:04:08,480 --> 00:04:10,189 they could potentially read back your 126 00:04:10,190 --> 00:04:12,559 private key in the case of Tory 127 00:04:12,560 --> 00:04:14,599 might feel to look at a relay and look at 128 00:04:14,600 --> 00:04:15,949 other information that is currently being 129 00:04:15,950 --> 00:04:17,989 passed across that relay or one of its 130 00:04:17,990 --> 00:04:19,669 short term keys. 131 00:04:19,670 --> 00:04:21,739 So it's affected a large number of 132 00:04:21,740 --> 00:04:23,179 of different services. 133 00:04:23,180 --> 00:04:24,679 We're going to focus on Websters 134 00:04:24,680 --> 00:04:26,029 primarily because that's where we saw a 135 00:04:26,030 --> 00:04:27,949 lot going on. 136 00:04:27,950 --> 00:04:29,569 But it did affect everything from mail 137 00:04:29,570 --> 00:04:31,399 servers to database servers to everything 138 00:04:31,400 --> 00:04:32,400 else. 139 00:04:34,300 --> 00:04:35,859 So when this when this vulnerability 140 00:04:35,860 --> 00:04:37,269 first came out, we decided to try to 141 00:04:37,270 --> 00:04:39,039 track it's essentially global 142 00:04:40,060 --> 00:04:41,559 effect on the entire Internet, and the 143 00:04:41,560 --> 00:04:43,629 way we did this is by scanning the 144 00:04:43,630 --> 00:04:44,919 entire Internet to look for the 145 00:04:44,920 --> 00:04:46,999 vulnerability, by altering the ZENAB 146 00:04:47,000 --> 00:04:49,279 scanner and scanning the IPV for address 147 00:04:49,280 --> 00:04:50,529 space on a regular basis. 148 00:04:50,530 --> 00:04:53,169 Scandi Alexa top one million websites, 149 00:04:53,170 --> 00:04:55,269 scanning four different mail servers in 150 00:04:55,270 --> 00:04:56,979 four different database servers and just 151 00:04:56,980 --> 00:04:58,959 looking at who is vulnerable. 152 00:04:58,960 --> 00:05:01,119 But we do this fairly special way. 153 00:05:01,120 --> 00:05:02,649 We didn't want to take advantage of the 154 00:05:02,650 --> 00:05:04,359 actual vulnerability itself. 155 00:05:04,360 --> 00:05:06,729 We didn't want to leak any private data 156 00:05:06,730 --> 00:05:08,649 from other people's servers. 157 00:05:08,650 --> 00:05:10,749 And so instead of actually exploiting 158 00:05:10,750 --> 00:05:12,819 vulnerability, we took advantage 159 00:05:12,820 --> 00:05:14,739 of a rather lucky phenomenon that was 160 00:05:14,740 --> 00:05:16,659 associated with a specific vulnerable 161 00:05:16,660 --> 00:05:18,729 version of open SSL. 162 00:05:18,730 --> 00:05:20,859 And according to the RC, in 163 00:05:20,860 --> 00:05:23,259 which the extension is defined, 164 00:05:23,260 --> 00:05:25,449 essentially says, if I send you a zero 165 00:05:25,450 --> 00:05:27,009 byte request that doesn't have any 166 00:05:27,010 --> 00:05:29,199 information to be echoed back to me, 167 00:05:29,200 --> 00:05:31,089 you should drop that request. 168 00:05:31,090 --> 00:05:32,679 And there's really nothing that's just 169 00:05:32,680 --> 00:05:34,749 not current staff. See, it's not quite 170 00:05:34,750 --> 00:05:36,219 a spec, just drop it. 171 00:05:36,220 --> 00:05:38,199 But in the specific vulnerable version of 172 00:05:38,200 --> 00:05:40,239 open SSL, instead of dropping that 173 00:05:40,240 --> 00:05:41,949 request to actually respond back with 174 00:05:41,950 --> 00:05:43,809 some random patting, it would respond 175 00:05:43,810 --> 00:05:46,149 back with any data, but you'd get back 176 00:05:46,150 --> 00:05:47,829 or a response back. 177 00:05:47,830 --> 00:05:49,929 And only the one specific 178 00:05:49,930 --> 00:05:52,329 vulnerable version of open SSL 179 00:05:52,330 --> 00:05:54,159 acted like this. The patched versions of 180 00:05:54,160 --> 00:05:56,229 open SSL actually fix this bug in 181 00:05:56,230 --> 00:05:58,159 different libraries. 182 00:05:58,160 --> 00:06:00,309 Can you tell us, for example, all file 183 00:06:00,310 --> 00:06:01,869 RACC and didn't have this specific 184 00:06:01,870 --> 00:06:03,819 phenomenon? So essentially what this let 185 00:06:03,820 --> 00:06:05,799 us do is that let's check whether a 186 00:06:05,800 --> 00:06:07,539 server was vulnerable without ever 187 00:06:07,540 --> 00:06:09,579 exploiting the actual Heartbleed 188 00:06:09,580 --> 00:06:10,580 vulnerability. 189 00:06:13,660 --> 00:06:15,399 So probably what's the most interesting 190 00:06:15,400 --> 00:06:17,919 people are the top 100 websites 191 00:06:17,920 --> 00:06:19,419 in the top 100 websites actually did 192 00:06:19,420 --> 00:06:21,699 fairly well at patching 193 00:06:21,700 --> 00:06:23,859 all of the top 500 websites from 194 00:06:23,860 --> 00:06:25,299 within the first forty eight hours when 195 00:06:25,300 --> 00:06:26,709 we started to do our scanning. 196 00:06:27,790 --> 00:06:29,029 When we can go back and look. 197 00:06:29,030 --> 00:06:30,279 However, though, because these are very 198 00:06:30,280 --> 00:06:31,929 large organizations are kind of in the 199 00:06:31,930 --> 00:06:34,149 press and media spotlight, most 200 00:06:34,150 --> 00:06:36,549 of them put out press releases. 201 00:06:36,550 --> 00:06:38,439 They answered questions from reporters. 202 00:06:38,440 --> 00:06:39,849 And by going through interrogating these 203 00:06:39,850 --> 00:06:41,559 pieces of data, we found that about forty 204 00:06:41,560 --> 00:06:43,059 four of the top 100 websites were 205 00:06:43,060 --> 00:06:44,199 initially vulnerable. 206 00:06:44,200 --> 00:06:45,789 So a fairly large percentage. 207 00:06:45,790 --> 00:06:47,229 And this is a lower bound. 208 00:06:47,230 --> 00:06:49,209 There are some organizations that never 209 00:06:49,210 --> 00:06:50,649 responded who didn't really 210 00:06:52,720 --> 00:06:54,609 didn't put out any information, but these 211 00:06:54,610 --> 00:06:56,109 were the numbers that did information 212 00:06:56,110 --> 00:06:58,209 that they were vulnerable and a handful 213 00:06:58,210 --> 00:06:59,589 of websites were so vulnerable. 214 00:06:59,590 --> 00:07:01,779 Twenty four hours, this included 215 00:07:01,780 --> 00:07:04,179 Yahoo! Imja, Stack Overflow. 216 00:07:04,180 --> 00:07:05,589 And these were the sites that we saw 217 00:07:05,590 --> 00:07:06,769 active attacks going on. 218 00:07:06,770 --> 00:07:07,899 People were stealing Yahoo! 219 00:07:07,900 --> 00:07:09,399 Credentials for Yahoo! 220 00:07:09,400 --> 00:07:11,529 Mail accounts for essentially in 221 00:07:11,530 --> 00:07:13,779 the wild 24 hours after 222 00:07:13,780 --> 00:07:15,699 the vulnerability was released. 223 00:07:15,700 --> 00:07:17,409 Within forty eight hours, all of these 224 00:07:17,410 --> 00:07:18,410 had been patched. 225 00:07:20,350 --> 00:07:22,599 Over the top 100 websites are fairly 226 00:07:22,600 --> 00:07:24,759 special, these are websites that 227 00:07:24,760 --> 00:07:26,379 probably not only have a dedicated 228 00:07:26,380 --> 00:07:28,059 sysadmin, they might have dedicated teams 229 00:07:28,060 --> 00:07:29,889 of sysadmins who are looking after these 230 00:07:29,890 --> 00:07:32,199 sites. What about the other websites? 231 00:07:32,200 --> 00:07:34,059 In our first scan, we've found our first 232 00:07:34,060 --> 00:07:35,709 scan approximately two days after the 233 00:07:35,710 --> 00:07:37,779 vulnerability was first released in 234 00:07:37,780 --> 00:07:39,849 about 48 hours after the 235 00:07:39,850 --> 00:07:41,409 first thing we found is that about 45 236 00:07:41,410 --> 00:07:43,989 percent of all Web sites for EDPS. 237 00:07:43,990 --> 00:07:46,239 And that's a bit discouraging 238 00:07:47,710 --> 00:07:49,209 to start. 239 00:07:49,210 --> 00:07:51,759 But of those approximately 60 percent of 240 00:07:51,760 --> 00:07:53,709 sites, the heartbeat. 241 00:07:53,710 --> 00:07:54,669 But this doesn't mean that they were 242 00:07:54,670 --> 00:07:55,909 initially vulnerable. 243 00:07:55,910 --> 00:07:57,969 Other pieces of software such as Tlas 244 00:07:57,970 --> 00:08:00,249 or Microsoft is supported 245 00:08:00,250 --> 00:08:02,019 the extension, but they didn't have this 246 00:08:02,020 --> 00:08:04,299 bug, so they weren't necessarily 247 00:08:04,300 --> 00:08:06,609 vulnerable like open SSL was. 248 00:08:06,610 --> 00:08:08,979 Well, if we split this out even further, 249 00:08:08,980 --> 00:08:11,139 we start to look that approximately 250 00:08:11,140 --> 00:08:13,389 91 percent of the sites used open 251 00:08:13,390 --> 00:08:15,489 SSL to facilitate connections. 252 00:08:15,490 --> 00:08:17,169 So we were able to look at this based on 253 00:08:17,170 --> 00:08:18,369 the server agent strings that are 254 00:08:18,370 --> 00:08:19,870 reported back by Web servers. 255 00:08:21,190 --> 00:08:22,689 So nine percent are using these other 256 00:08:22,690 --> 00:08:24,130 software vendors, such as Microsoft. 257 00:08:25,300 --> 00:08:26,529 But this lets us estimate that 258 00:08:26,530 --> 00:08:28,689 approximately 55 percent of 259 00:08:28,690 --> 00:08:30,849 Web sites at most were likely 260 00:08:30,850 --> 00:08:33,158 vulnerable when the heartbeat 261 00:08:33,159 --> 00:08:34,929 vulnerability was first announced. 262 00:08:37,169 --> 00:08:39,058 So that's provides an upper bound on the 263 00:08:39,059 --> 00:08:40,979 lower bound, we're able to estimate as 264 00:08:40,980 --> 00:08:43,229 well, because two extensions are 265 00:08:43,230 --> 00:08:44,729 two pieces of functionality were also 266 00:08:44,730 --> 00:08:46,559 introduced along with the heartbeat 267 00:08:46,560 --> 00:08:48,809 extension and open SSL one, that 268 00:08:48,810 --> 00:08:50,099 one, 2012. 269 00:08:50,100 --> 00:08:51,659 And these are tell us one point one, two, 270 00:08:51,660 --> 00:08:53,339 one point two. And so while no one is 271 00:08:53,340 --> 00:08:54,659 really thinking about the heartbeat 272 00:08:54,660 --> 00:08:56,339 extension, no one was really testing for 273 00:08:56,340 --> 00:08:58,349 it before the Heartbleed vulnerability 274 00:08:58,350 --> 00:09:00,419 happened, people were checking for 275 00:09:00,420 --> 00:09:01,769 less one point one at one point two. 276 00:09:01,770 --> 00:09:03,299 These are major features that our people 277 00:09:03,300 --> 00:09:06,069 are interested in, the deployment of 278 00:09:06,070 --> 00:09:08,369 an even Ristic, who runs the SSL 279 00:09:08,370 --> 00:09:10,629 pulse and the Quarless SSL 280 00:09:10,630 --> 00:09:11,879 lab site where everyone goes to check 281 00:09:11,880 --> 00:09:13,979 their their website to see if it's 282 00:09:13,980 --> 00:09:15,989 correctly configured, have actually been 283 00:09:15,990 --> 00:09:17,969 scanning for this and for looking for the 284 00:09:17,970 --> 00:09:20,429 deployment of these two new protocols. 285 00:09:20,430 --> 00:09:22,139 And what we found is that about thirty 286 00:09:22,140 --> 00:09:24,359 two percent of websites supported 287 00:09:24,360 --> 00:09:26,939 us one point one or one point two prior 288 00:09:26,940 --> 00:09:28,889 to the Heartbleed vulnerability. 289 00:09:28,890 --> 00:09:30,629 And that lets us estimate that 290 00:09:30,630 --> 00:09:32,399 approximately twenty four percent at a 291 00:09:32,400 --> 00:09:34,949 minimum of the web of websites 292 00:09:34,950 --> 00:09:37,169 that use GPS were initially vulnerable. 293 00:09:37,170 --> 00:09:38,939 So it lets us ballpark between twenty 294 00:09:38,940 --> 00:09:40,919 four to fifty five percent. 295 00:09:40,920 --> 00:09:42,149 In the beginning, a lot of the news 296 00:09:42,150 --> 00:09:43,979 reports said that somewhere between sixty 297 00:09:43,980 --> 00:09:46,499 sixty five percent were vulnerable 298 00:09:46,500 --> 00:09:48,179 just based on how many sites to use and 299 00:09:48,180 --> 00:09:49,919 Gen X or Apache. 300 00:09:49,920 --> 00:09:51,509 It might have been a little bit of a high 301 00:09:51,510 --> 00:09:53,249 guess, but not by much. 302 00:09:53,250 --> 00:09:54,779 A very large percent of the Internet was 303 00:09:54,780 --> 00:09:55,780 actually vulnerable. 304 00:09:57,620 --> 00:09:59,569 When we go back out, one more layer and 305 00:09:59,570 --> 00:10:01,489 look at the full IPV for address space, 306 00:10:01,490 --> 00:10:03,679 we see yet another story and we see 307 00:10:03,680 --> 00:10:06,199 that only approximately 11 percent of 308 00:10:06,200 --> 00:10:08,119 hosts support heartbeat. 309 00:10:08,120 --> 00:10:10,189 Six percent of hosts are vulnerable. 310 00:10:10,190 --> 00:10:12,409 And the main reason here is because most 311 00:10:12,410 --> 00:10:14,749 of the hosts on the IPV for address 312 00:10:14,750 --> 00:10:16,579 space are websites. 313 00:10:16,580 --> 00:10:18,109 They're not the Googles or the Yahoos. 314 00:10:18,110 --> 00:10:19,819 What they are of these embedded devices 315 00:10:19,820 --> 00:10:21,919 on these are home routers, their video 316 00:10:21,920 --> 00:10:24,049 cameras or these devices we'd hoped 317 00:10:24,050 --> 00:10:25,699 would never see the light of day and 318 00:10:25,700 --> 00:10:27,349 everything in between. 319 00:10:27,350 --> 00:10:29,419 And for the most part, these tend to use 320 00:10:29,420 --> 00:10:32,329 much slimmed down pieces of 321 00:10:32,330 --> 00:10:33,859 software. They never support the 322 00:10:33,860 --> 00:10:35,989 heartbeat extension to start with, but 323 00:10:35,990 --> 00:10:37,999 about six percent of hosts were 324 00:10:38,000 --> 00:10:38,899 vulnerable in. 325 00:10:38,900 --> 00:10:40,039 The sad thing about these embedded 326 00:10:40,040 --> 00:10:41,629 devices is almost all of the embedded 327 00:10:41,630 --> 00:10:43,400 devices still remain vulnerable today. 328 00:10:45,000 --> 00:10:46,609 We know that embedded devices aren't 329 00:10:46,610 --> 00:10:48,289 really patched, even if patches that are 330 00:10:48,290 --> 00:10:50,209 available in most cases they aren't. 331 00:10:50,210 --> 00:10:51,710 Users don't call and apply these 332 00:10:53,510 --> 00:10:54,679 in these range from everything. 333 00:10:54,680 --> 00:10:56,569 We really did see scattered devices that 334 00:10:56,570 --> 00:10:57,919 were vulnerable to Heartbleed. 335 00:10:57,920 --> 00:11:00,019 We even saw a thing with the strangest 336 00:11:00,020 --> 00:11:02,749 case was a pizza point of sale 337 00:11:02,750 --> 00:11:04,849 software that we saw repeatedly 338 00:11:04,850 --> 00:11:05,850 being vulnerable. 339 00:11:08,940 --> 00:11:10,109 When we start to look at the patching 340 00:11:10,110 --> 00:11:12,179 behavior, we actually did fairly well in 341 00:11:12,180 --> 00:11:14,459 terms of the top 100 Web sites or top 342 00:11:14,460 --> 00:11:16,139 1000000 websites, if we expected that 343 00:11:16,140 --> 00:11:18,329 between twenty four to fifty five percent 344 00:11:18,330 --> 00:11:20,429 of Web sites for vulnerable. 345 00:11:20,430 --> 00:11:21,959 When the vulnerability was first 346 00:11:21,960 --> 00:11:23,519 released, we were down to approximately 347 00:11:23,520 --> 00:11:25,979 11 to 12 percent within the first 48 348 00:11:25,980 --> 00:11:28,109 hours. And that's fairly impressive. 349 00:11:28,110 --> 00:11:29,939 We did pretty good as an operator 350 00:11:29,940 --> 00:11:31,679 community of getting our websites 351 00:11:31,680 --> 00:11:32,680 patched. 352 00:11:33,300 --> 00:11:34,829 What's a little bit satyrs when we look 353 00:11:34,830 --> 00:11:36,749 at the IP for address space and also look 354 00:11:36,750 --> 00:11:38,189 where the Aleksa websites kind of 355 00:11:38,190 --> 00:11:39,629 stagnated. 356 00:11:39,630 --> 00:11:41,549 Somewhere around four percent of Web 357 00:11:41,550 --> 00:11:43,139 sites remain vulnerable for quite a 358 00:11:43,140 --> 00:11:45,209 while. We're down to somewhere around 359 00:11:45,210 --> 00:11:46,739 below four percent. And unfortunately, 360 00:11:46,740 --> 00:11:48,969 the number remains there today. 361 00:11:48,970 --> 00:11:49,859 These are big websites. 362 00:11:49,860 --> 00:11:52,049 These are the top one million websites 363 00:11:52,050 --> 00:11:53,519 of approximately four percent are still 364 00:11:53,520 --> 00:11:54,520 vulnerable today. 365 00:11:56,610 --> 00:11:57,989 The one thing that jumps out right away 366 00:11:57,990 --> 00:11:59,439 is what happened here. 367 00:11:59,440 --> 00:12:00,899 There's this large jump for you almost 368 00:12:00,900 --> 00:12:03,419 have the number of vulnerable 369 00:12:03,420 --> 00:12:05,339 devices on the entire Internet. 370 00:12:05,340 --> 00:12:06,719 What happened here is that there are a 371 00:12:06,720 --> 00:12:08,549 couple of very large Azis that are very 372 00:12:08,550 --> 00:12:10,259 large shared hosting providers. 373 00:12:10,260 --> 00:12:11,879 Every single one of their websites was 374 00:12:11,880 --> 00:12:13,769 vulnerable on the vulnerability came out. 375 00:12:13,770 --> 00:12:15,599 They patched every one of their virtual 376 00:12:15,600 --> 00:12:17,999 machines and the number drops drastically 377 00:12:18,000 --> 00:12:19,049 in a couple of days. And these are 378 00:12:19,050 --> 00:12:20,699 actually you can kind of see there are 379 00:12:20,700 --> 00:12:22,619 three or four specific hosting providers 380 00:12:22,620 --> 00:12:24,359 all patched in this time period. 381 00:12:24,360 --> 00:12:26,519 But as you subtract that one, drop 382 00:12:26,520 --> 00:12:28,679 the IP before address pace itself 383 00:12:28,680 --> 00:12:30,359 is actually fairly stagnant. 384 00:12:31,560 --> 00:12:32,609 We haven't really patched it. 385 00:12:34,780 --> 00:12:37,509 So is this fast enough, 386 00:12:37,510 --> 00:12:39,279 we said twenty four to fifty five percent 387 00:12:39,280 --> 00:12:40,809 or initially vulnerable and this looks 388 00:12:40,810 --> 00:12:42,609 pretty good for the top one million 389 00:12:42,610 --> 00:12:44,709 websites. And when we said, are we 390 00:12:44,710 --> 00:12:46,719 fast enough to look at when did attacks 391 00:12:46,720 --> 00:12:47,769 start to first happen 392 00:12:48,970 --> 00:12:51,069 and sadly tech 393 00:12:51,070 --> 00:12:51,669 start to happen? 394 00:12:51,670 --> 00:12:54,129 Before we started to do our scans, 395 00:12:54,130 --> 00:12:56,379 we saw our first Internet wide scan 396 00:12:56,380 --> 00:12:57,729 for the Heartbleed vulnerability 397 00:12:57,730 --> 00:13:00,189 approximately twenty two hours 398 00:13:00,190 --> 00:13:01,839 after the vulnerability was released. 399 00:13:03,440 --> 00:13:05,539 So, yes, we did well, but we did not 400 00:13:05,540 --> 00:13:07,369 do quite good enough, and this is an 401 00:13:07,370 --> 00:13:08,959 Internet wide scan, if we consider 402 00:13:08,960 --> 00:13:11,089 targeted scans, these are happening much 403 00:13:11,090 --> 00:13:12,679 before. If you looked at who is attacking 404 00:13:12,680 --> 00:13:14,209 Yahoo! This is going to happen before 405 00:13:14,210 --> 00:13:16,219 these large, indiscriminate scans of the 406 00:13:16,220 --> 00:13:17,840 full IP for address space. 407 00:13:18,950 --> 00:13:20,989 So looking at the entire attack scene, 408 00:13:20,990 --> 00:13:22,639 what we did is we looked at a honeypot on 409 00:13:22,640 --> 00:13:23,840 the Amazon. Easy to 410 00:13:25,790 --> 00:13:27,829 just see an unused IP address that was 411 00:13:27,830 --> 00:13:29,329 watching all traffic that came in. 412 00:13:29,330 --> 00:13:31,429 We also looked at packet traces from 413 00:13:31,430 --> 00:13:33,619 the Lawrence Berkeley National Lab and 414 00:13:33,620 --> 00:13:35,989 from EXI in Berkeley 415 00:13:35,990 --> 00:13:37,789 to look at what techs were coming into 416 00:13:37,790 --> 00:13:40,369 these three disparate networks. 417 00:13:40,370 --> 00:13:42,379 And we find no evidence of attacks taking 418 00:13:42,380 --> 00:13:44,269 place prior to disclosure. 419 00:13:44,270 --> 00:13:45,469 And this doesn't mean that no one knew 420 00:13:45,470 --> 00:13:46,969 about it, but it means that no one was 421 00:13:46,970 --> 00:13:48,739 doing large, indiscriminate Internet wide 422 00:13:48,740 --> 00:13:50,239 scans of it. There may have been targeted 423 00:13:50,240 --> 00:13:52,189 attacks, but we weren't seeing large 424 00:13:52,190 --> 00:13:53,659 exploitations in the wild. 425 00:13:54,680 --> 00:13:56,569 The first piece of scanner traffic we saw 426 00:13:56,570 --> 00:13:58,129 was approximately two hours after 427 00:13:58,130 --> 00:14:00,169 disclosure and originated from the 428 00:14:00,170 --> 00:14:01,549 University of Latvia. 429 00:14:01,550 --> 00:14:03,289 And in total, we observed approximately 430 00:14:03,290 --> 00:14:05,359 six thousand different Heartbleed exploit 431 00:14:05,360 --> 00:14:07,519 attempts from almost 700 432 00:14:07,520 --> 00:14:09,619 hosts within that first couple of 433 00:14:09,620 --> 00:14:12,229 weeks. And there are two major outliers 434 00:14:12,230 --> 00:14:13,579 that jump out right away, which are 435 00:14:13,580 --> 00:14:15,649 Philippos site that lets 436 00:14:15,650 --> 00:14:17,329 you test whether your website was 437 00:14:17,330 --> 00:14:18,559 vulnerable to Heartbleed. 438 00:14:18,560 --> 00:14:20,419 And the second was SSL Labs, which had a 439 00:14:20,420 --> 00:14:22,249 similar app that let you test for the 440 00:14:22,250 --> 00:14:23,250 vulnerability. 441 00:14:24,230 --> 00:14:25,579 And what we find is actually the number 442 00:14:25,580 --> 00:14:27,289 of people doing large Internet wide scans 443 00:14:27,290 --> 00:14:28,849 was actually fairly small. 444 00:14:28,850 --> 00:14:31,169 Only 11 hosken both the Amazon, 445 00:14:31,170 --> 00:14:33,289 easy to honeypot and our 446 00:14:33,290 --> 00:14:35,359 network at Berkeley and 447 00:14:35,360 --> 00:14:37,189 approximately only six percent total 448 00:14:37,190 --> 00:14:38,389 actually scanned more than one hundred 449 00:14:38,390 --> 00:14:40,129 hosts. So there actually appeared to be 450 00:14:40,130 --> 00:14:42,259 very little Internet wide scanning. 451 00:14:42,260 --> 00:14:43,189 Only six hosts. 452 00:14:43,190 --> 00:14:44,569 And several of these are academic 453 00:14:44,570 --> 00:14:46,819 institutions, Michigan and to 454 00:14:46,820 --> 00:14:49,699 Berlin, that were looking 455 00:14:49,700 --> 00:14:51,289 for the vulnerability. 456 00:14:51,290 --> 00:14:53,599 There were PSol, however, for other 457 00:14:53,600 --> 00:14:55,909 hosts to in 458 00:14:55,910 --> 00:14:57,739 Chinese As-Is that were completely 459 00:14:57,740 --> 00:14:58,679 unidentifiable. 460 00:14:58,680 --> 00:15:00,109 We really don't know what they were 461 00:15:00,110 --> 00:15:01,009 looking for. They could have been 462 00:15:01,010 --> 00:15:02,189 academic institutions. 463 00:15:02,190 --> 00:15:03,169 They could have been attackers. 464 00:15:03,170 --> 00:15:04,909 We really no way of telling. 465 00:15:04,910 --> 00:15:06,199 We know that they actually exploited the 466 00:15:06,200 --> 00:15:07,670 vulnerability, but that's about it. 467 00:15:10,460 --> 00:15:12,109 On the other hand, there are a very large 468 00:15:12,110 --> 00:15:14,059 number of hosts that targeted our 469 00:15:14,060 --> 00:15:15,829 honeypot. And this wasn't a well-known 470 00:15:15,830 --> 00:15:18,019 website, but it does show 471 00:15:18,020 --> 00:15:19,999 that with almost 200 hosts scanning that 472 00:15:20,000 --> 00:15:21,709 honeypot, people, attackers are going 473 00:15:21,710 --> 00:15:23,079 after this other address space. 474 00:15:23,080 --> 00:15:25,669 They are looking after these possibly 475 00:15:25,670 --> 00:15:26,899 their cloud providers. 476 00:15:26,900 --> 00:15:28,579 They could possibly just be more densely 477 00:15:28,580 --> 00:15:30,649 populated IP address space. 478 00:15:33,430 --> 00:15:35,649 Two weeks after disclosure, approximately 479 00:15:35,650 --> 00:15:37,059 six hundred thousand hosts on the 480 00:15:37,060 --> 00:15:38,949 Internet remain vulnerable, and at this 481 00:15:38,950 --> 00:15:41,079 time we decided to go ahead and to 482 00:15:41,080 --> 00:15:43,359 notify everyone on the Internet who 483 00:15:43,360 --> 00:15:44,379 had a vulnerable host. 484 00:15:46,030 --> 00:15:48,399 And this is one of the first times 485 00:15:48,400 --> 00:15:50,469 one of these large kind of Internet 486 00:15:50,470 --> 00:15:52,539 wide vulnerability disclosures had been 487 00:15:52,540 --> 00:15:53,949 done, so we didn't necessarily know how 488 00:15:53,950 --> 00:15:55,359 this would go. 489 00:15:55,360 --> 00:15:57,459 And so we decided to run an 490 00:15:57,460 --> 00:15:59,529 experiment and essentially split these 491 00:15:59,530 --> 00:16:01,419 hosts into two groups and to see what was 492 00:16:01,420 --> 00:16:03,069 the effect of actually doing an Internet 493 00:16:03,070 --> 00:16:04,499 wide notification. 494 00:16:04,500 --> 00:16:06,549 And so we contacted everyone eventually 495 00:16:06,550 --> 00:16:08,319 that had a vulnerable host, but we split 496 00:16:08,320 --> 00:16:10,329 them into two groups and contacted them 497 00:16:10,330 --> 00:16:12,309 approximately two weeks apart. 498 00:16:12,310 --> 00:16:13,539 And what we find is actually quite 499 00:16:13,540 --> 00:16:15,399 surprising for the research community. 500 00:16:15,400 --> 00:16:17,079 When we went into this, we expected that 501 00:16:17,080 --> 00:16:18,519 global notifications would have actually 502 00:16:18,520 --> 00:16:20,079 very little impact on actually who 503 00:16:20,080 --> 00:16:21,080 patched. 504 00:16:21,850 --> 00:16:23,439 And for a long time we've believed this. 505 00:16:23,440 --> 00:16:24,519 And what we found was that actually the 506 00:16:24,520 --> 00:16:26,229 opposite. We found that by doing these 507 00:16:26,230 --> 00:16:27,759 global notifications, we could actually 508 00:16:27,760 --> 00:16:30,859 increase patching by almost 50 percent. 509 00:16:30,860 --> 00:16:32,199 And this is actually fairly encouraging 510 00:16:32,200 --> 00:16:33,489 to us as researchers. 511 00:16:33,490 --> 00:16:35,589 I think a lot of times when we find 512 00:16:35,590 --> 00:16:37,449 vulnerabilities, we say there's nothing 513 00:16:37,450 --> 00:16:38,529 we can do. 514 00:16:38,530 --> 00:16:39,729 We found these, but no one will pay 515 00:16:39,730 --> 00:16:40,929 attention. 516 00:16:40,930 --> 00:16:42,729 And what we found is that actually people 517 00:16:42,730 --> 00:16:44,049 are paying attention. 518 00:16:44,050 --> 00:16:45,519 Even if you don't get responses to 519 00:16:45,520 --> 00:16:47,919 emails, even if we don't get a positive 520 00:16:47,920 --> 00:16:50,319 response back, people are actually 521 00:16:50,320 --> 00:16:52,659 reading these and actually are 522 00:16:53,680 --> 00:16:55,599 catching even in cases where we received 523 00:16:55,600 --> 00:16:57,369 bounce notifications back and we thought 524 00:16:57,370 --> 00:16:59,379 no one actually received this email on. 525 00:16:59,380 --> 00:17:00,759 A lot of times the email made it through 526 00:17:00,760 --> 00:17:02,589 to a second person that we didn't know 527 00:17:02,590 --> 00:17:04,749 about. And even though it looks like 528 00:17:04,750 --> 00:17:06,279 a wasted effort to us to actually 529 00:17:06,280 --> 00:17:08,199 increase the trace, just something to 530 00:17:08,200 --> 00:17:10,449 keep in mind as you're doing these large 531 00:17:10,450 --> 00:17:11,559 Internet wide scams. 532 00:17:13,550 --> 00:17:15,409 But unfortunately, patching wasn't 533 00:17:15,410 --> 00:17:17,449 enough, cryptographic keys were also at 534 00:17:17,450 --> 00:17:18,450 risk. 535 00:17:24,520 --> 00:17:27,108 So partly let you steal cryptographic 536 00:17:27,109 --> 00:17:29,169 keys to talk about this a little bit 537 00:17:29,170 --> 00:17:31,179 in the second talk, but essentially if 538 00:17:31,180 --> 00:17:33,879 you are using GPS or Apache, overengineer 539 00:17:33,880 --> 00:17:34,839 someone could go and they could 540 00:17:34,840 --> 00:17:36,729 potentially pull down your cryptographic 541 00:17:36,730 --> 00:17:38,499 key. And so not only did you need to 542 00:17:38,500 --> 00:17:40,479 patch your server, you needed to change 543 00:17:40,480 --> 00:17:41,889 your cryptographic key. 544 00:17:41,890 --> 00:17:44,529 And while we did fairly well, patching 545 00:17:44,530 --> 00:17:46,539 the sad story is the cryptographic keys 546 00:17:46,540 --> 00:17:47,619 didn't go nearly as well. 547 00:17:49,120 --> 00:17:51,309 What we find is that only 10 percent 548 00:17:51,310 --> 00:17:53,049 of the sites we found vulnerable actually 549 00:17:53,050 --> 00:17:54,310 replace their certificate's. 550 00:17:56,050 --> 00:17:58,389 Even more discouraging, 14 percent 551 00:17:58,390 --> 00:17:59,739 of the websites that did replace replacer 552 00:17:59,740 --> 00:18:01,599 certificates just reuse the same 553 00:18:01,600 --> 00:18:02,650 vulnerable private key. 554 00:18:08,170 --> 00:18:09,609 So no protection 555 00:18:12,430 --> 00:18:13,899 for percent of websites that were 556 00:18:13,900 --> 00:18:15,550 vulnerable revoke their certificates. 557 00:18:17,740 --> 00:18:19,809 These are pretty dismal numbers. 558 00:18:19,810 --> 00:18:21,070 There's a lot of improvement here. 559 00:18:23,150 --> 00:18:25,219 So what do we learn out of all of this? 560 00:18:26,810 --> 00:18:28,339 I think the first thing that jumps out at 561 00:18:28,340 --> 00:18:30,589 us is that we have these large 562 00:18:30,590 --> 00:18:32,089 open source projects that we are all 563 00:18:32,090 --> 00:18:34,039 depending on. We have open SSL. 564 00:18:34,040 --> 00:18:36,139 We have 60 percent of websites 565 00:18:36,140 --> 00:18:37,839 that are depending on this for a 566 00:18:39,680 --> 00:18:41,299 lack of attention. Support to these 567 00:18:41,300 --> 00:18:43,339 critical projects is likely what led to 568 00:18:43,340 --> 00:18:45,259 these types of massive vulnerabilities. 569 00:18:45,260 --> 00:18:46,909 These projects deserve more attention 570 00:18:46,910 --> 00:18:49,099 from us. They deserve more support from 571 00:18:49,100 --> 00:18:50,100 us. 572 00:18:58,890 --> 00:19:01,019 The ecosystem is incredibly 573 00:19:01,020 --> 00:19:02,309 fragile, fragile. 574 00:19:02,310 --> 00:19:03,839 I wish we could say a Heartbleed was the 575 00:19:03,840 --> 00:19:05,669 only major vulnerability we had last 576 00:19:05,670 --> 00:19:07,349 year. It was one of several. 577 00:19:09,270 --> 00:19:11,549 If we look at poodle, if we look 578 00:19:11,550 --> 00:19:13,319 at its channel, I know pools a little bit 579 00:19:13,320 --> 00:19:15,719 less severe than than 580 00:19:15,720 --> 00:19:17,429 as a channel or Heartbleed, but we 581 00:19:17,430 --> 00:19:19,559 affected almost every site between 582 00:19:19,560 --> 00:19:21,629 those two vulnerabilities. 583 00:19:21,630 --> 00:19:23,939 One was a remote code exploitation. 584 00:19:23,940 --> 00:19:25,199 The other one allowed us to steal 585 00:19:25,200 --> 00:19:28,109 credentials and cryptographic keys. 586 00:19:28,110 --> 00:19:29,639 These are bad, right? 587 00:19:29,640 --> 00:19:30,640 These are pretty severe. 588 00:19:31,620 --> 00:19:33,659 These continue to come. 589 00:19:33,660 --> 00:19:35,819 We just saw Puddle to a matter of 590 00:19:35,820 --> 00:19:36,820 weeks ago. 591 00:19:38,580 --> 00:19:39,839 Remains incredibly fragile. 592 00:19:39,840 --> 00:19:42,329 Even today, the supporting 593 00:19:42,330 --> 00:19:44,339 Pecky is not equipped to handle the 594 00:19:44,340 --> 00:19:45,389 massive revocation. 595 00:19:45,390 --> 00:19:46,529 I'm Nick. We'll talk about this a little 596 00:19:46,530 --> 00:19:48,479 bit more. But essentially the current 597 00:19:48,480 --> 00:19:49,579 what we have in place, which is 598 00:19:49,580 --> 00:19:51,269 certificate revocation, does not work 599 00:19:51,270 --> 00:19:53,459 right now, no matter the fact 600 00:19:53,460 --> 00:19:55,409 that only four percent of websites tried 601 00:19:55,410 --> 00:19:57,329 to revoke their certificates, the 602 00:19:57,330 --> 00:19:59,010 ecosystem barely handled that. 603 00:20:00,180 --> 00:20:02,219 We could say actually didn't. 604 00:20:02,220 --> 00:20:04,049 Web browsers were not aware of most of 605 00:20:04,050 --> 00:20:05,759 the revocations that took place. 606 00:20:05,760 --> 00:20:07,439 As we start to discuss and we have then 607 00:20:07,440 --> 00:20:08,909 as a community discussing how do we 608 00:20:08,910 --> 00:20:11,189 change revocation or what we do going 609 00:20:11,190 --> 00:20:12,989 forward, we need to consider the fact 610 00:20:12,990 --> 00:20:15,149 that massive fraud vacation is something 611 00:20:15,150 --> 00:20:16,829 that we have to take account for. 612 00:20:16,830 --> 00:20:18,359 We need to take account for the fact that 613 00:20:18,360 --> 00:20:20,039 we might need to revoke every certificate 614 00:20:20,040 --> 00:20:21,299 on the Internet. 615 00:20:21,300 --> 00:20:22,509 That's what happened here. 616 00:20:22,510 --> 00:20:24,329 We're not talking about trickle here. 617 00:20:24,330 --> 00:20:25,679 They're one person lost their private 618 00:20:25,680 --> 00:20:27,609 key. We need to be able to handle large 619 00:20:27,610 --> 00:20:28,770 correlated revocation. 620 00:20:30,490 --> 00:20:32,259 The recent advances in scanning let us 621 00:20:32,260 --> 00:20:34,239 quickly respond and measure the impact 622 00:20:34,240 --> 00:20:35,559 here from one of the first times you 623 00:20:35,560 --> 00:20:37,149 could actually see what was the impact of 624 00:20:37,150 --> 00:20:39,649 one of these large vulnerabilities in 625 00:20:39,650 --> 00:20:40,929 in a positive light. 626 00:20:40,930 --> 00:20:43,089 Let us actually not only understand what 627 00:20:43,090 --> 00:20:44,409 happened, but impact patching. 628 00:20:44,410 --> 00:20:46,419 We're able to increase patching for the 629 00:20:46,420 --> 00:20:48,729 entire Internet by almost 50 percent. 630 00:20:48,730 --> 00:20:50,499 And that's incredibly encouraging. 631 00:20:50,500 --> 00:20:52,119 And we have this fragile ecosystem, but 632 00:20:52,120 --> 00:20:53,679 we're able to spur on patch and we're 633 00:20:53,680 --> 00:20:55,419 able to help out. 634 00:20:55,420 --> 00:20:56,739 But at the same time, we think we do have 635 00:20:56,740 --> 00:20:59,019 to take a look back and say 636 00:20:59,020 --> 00:21:00,730 there's a lot of room to grow still. 637 00:21:02,780 --> 00:21:04,339 And with that, and we'll switch over to 638 00:21:04,340 --> 00:21:05,340 Nick. 639 00:21:27,830 --> 00:21:29,809 OK, this seems to be working. 640 00:21:29,810 --> 00:21:31,189 Hello, everybody, I'm Nick. 641 00:21:31,190 --> 00:21:33,259 I'm the security engineering lead 642 00:21:33,260 --> 00:21:34,609 at CloudFlare. 643 00:21:34,610 --> 00:21:37,099 And I'm going to talk to you today about 644 00:21:37,100 --> 00:21:38,779 Heartbleed if you guys aren't sick of it 645 00:21:38,780 --> 00:21:40,969 already after this long 646 00:21:40,970 --> 00:21:43,309 year. So specifically, 647 00:21:43,310 --> 00:21:44,749 I going to talk about my personal 648 00:21:44,750 --> 00:21:47,209 experience or my experience at CloudFlare 649 00:21:47,210 --> 00:21:49,309 with what happened during 650 00:21:49,310 --> 00:21:51,889 during Heartbleed, when it came out 651 00:21:51,890 --> 00:21:55,009 and what happened after the fact. 652 00:21:55,010 --> 00:21:57,589 So has anybody read this 653 00:21:57,590 --> 00:21:58,639 article? 654 00:21:58,640 --> 00:22:01,039 This is this is a snippet from 655 00:22:01,040 --> 00:22:03,289 The Verge and it 656 00:22:03,290 --> 00:22:05,419 is kind of semi fictional account 657 00:22:05,420 --> 00:22:07,669 of what happened during the disclosure 658 00:22:07,670 --> 00:22:08,670 of of Heartbleed. 659 00:22:10,160 --> 00:22:12,229 I can kind of paraphrase a little bit 660 00:22:12,230 --> 00:22:14,359 how that conversation went, but 661 00:22:14,360 --> 00:22:15,919 it kind of kind of got a little bit like 662 00:22:15,920 --> 00:22:17,989 this. Like so do you do you 663 00:22:17,990 --> 00:22:19,290 use details? 664 00:22:20,510 --> 00:22:22,039 No, not that I know of. 665 00:22:22,040 --> 00:22:23,509 Does anybody use details? 666 00:22:23,510 --> 00:22:25,759 And that's for websites. 667 00:22:25,760 --> 00:22:26,789 Nobody use data. 668 00:22:26,790 --> 00:22:28,579 Grumped details. 669 00:22:28,580 --> 00:22:29,680 How about topics? 670 00:22:31,280 --> 00:22:33,559 Well, while my answer was 671 00:22:33,560 --> 00:22:35,119 what's Attila's heartbeat? 672 00:22:35,120 --> 00:22:37,399 I mean, at this point, barely anybody 673 00:22:37,400 --> 00:22:39,409 knew what this was. It was very obscure 674 00:22:39,410 --> 00:22:40,339 feature. 675 00:22:40,340 --> 00:22:42,439 And the answer 676 00:22:42,440 --> 00:22:44,659 here was, was this the stupid 677 00:22:44,660 --> 00:22:45,859 and and there's a bug. 678 00:22:45,860 --> 00:22:47,659 You should turn them off, right? 679 00:22:47,660 --> 00:22:49,909 So recompile open SSL 680 00:22:49,910 --> 00:22:50,899 with openness. 681 00:22:50,900 --> 00:22:52,069 Lascelles No heartbeat's. 682 00:22:52,070 --> 00:22:54,229 And that was, that was kind of that 683 00:22:54,230 --> 00:22:56,419 was that. So heartbeat's were off 684 00:22:56,420 --> 00:22:58,099 for CloudFlare. We have a pretty simple 685 00:22:58,100 --> 00:23:00,019 architecture so deployed it really 686 00:23:00,020 --> 00:23:02,149 quickly and the answer was 687 00:23:02,150 --> 00:23:03,379 OK, looks good. 688 00:23:03,380 --> 00:23:05,569 Public disclosure should be around April 689 00:23:05,570 --> 00:23:06,679 9th. 690 00:23:06,680 --> 00:23:08,839 So I guess a lot of people had this 691 00:23:08,840 --> 00:23:10,939 question. But why tell CloudFlare. 692 00:23:10,940 --> 00:23:13,069 Well, let me just go real quick 693 00:23:13,070 --> 00:23:15,769 and describe what CloudFlare does. 694 00:23:15,770 --> 00:23:18,409 So it's a reverse proxy. 695 00:23:18,410 --> 00:23:20,689 So if you have a website, 696 00:23:20,690 --> 00:23:22,669 CloudFlare can sit in front of it and 697 00:23:22,670 --> 00:23:23,779 block malicious traffic. 698 00:23:23,780 --> 00:23:26,269 That's the sort of red X up there, 699 00:23:26,270 --> 00:23:28,459 as well as send cached content, 700 00:23:28,460 --> 00:23:30,949 static content. That's the bright orange. 701 00:23:30,950 --> 00:23:33,079 So it reduces anybody getting 702 00:23:33,080 --> 00:23:35,359 to your website that's malicious 703 00:23:35,360 --> 00:23:37,909 and reduces the load on your website. 704 00:23:37,910 --> 00:23:40,219 And for this to work, this cloud 705 00:23:40,220 --> 00:23:41,809 has to be closer to the visitor than it 706 00:23:41,810 --> 00:23:43,309 does have to be to your website. 707 00:23:43,310 --> 00:23:45,559 So we have this global network. 708 00:23:45,560 --> 00:23:48,079 So are our nodes are closer to 709 00:23:48,080 --> 00:23:49,639 the visitors and that's that's how it 710 00:23:49,640 --> 00:23:51,859 works. And there's over a million 711 00:23:51,860 --> 00:23:53,899 sites on CloudFlare, including banks, 712 00:23:53,900 --> 00:23:56,149 government websites, Bitcoin exchanges, 713 00:23:56,150 --> 00:23:57,499 almost every Bitcoin exchanges on 714 00:23:57,500 --> 00:23:59,569 CloudFlare, the ETFs 715 00:23:59,570 --> 00:24:01,609 website, Reddit. 716 00:24:01,610 --> 00:24:03,769 I can go on and on and on. 717 00:24:03,770 --> 00:24:04,939 So lots of sites. 718 00:24:04,940 --> 00:24:07,249 But what CloudFlare does is very simple. 719 00:24:07,250 --> 00:24:09,439 It's essentially three 720 00:24:09,440 --> 00:24:12,169 services or two DNS, 721 00:24:12,170 --> 00:24:13,970 HTP and 722 00:24:15,050 --> 00:24:17,629 which is powered by open SSL 723 00:24:17,630 --> 00:24:19,009 and Engine X. 724 00:24:19,010 --> 00:24:21,079 So the architecture 725 00:24:21,080 --> 00:24:22,789 is very simple in that every machine that 726 00:24:22,790 --> 00:24:25,159 we have can serve every site. 727 00:24:25,160 --> 00:24:27,499 So thinking back to 728 00:24:27,500 --> 00:24:30,289 Heartbleed, this is actually 729 00:24:30,290 --> 00:24:32,029 you can see why this would be a really, 730 00:24:32,030 --> 00:24:34,429 really bad situation for Heartbleed 731 00:24:34,430 --> 00:24:36,859 to hit anyways. 732 00:24:36,860 --> 00:24:38,989 So it happened early 733 00:24:38,990 --> 00:24:40,309 April 7th. 734 00:24:40,310 --> 00:24:42,439 Ten, twenty seven open SSL publish 735 00:24:42,440 --> 00:24:44,659 their advisory and that hit 736 00:24:44,660 --> 00:24:46,549 Hacker News really quickly after that. 737 00:24:46,550 --> 00:24:48,649 Within half an hour it hit the front page 738 00:24:48,650 --> 00:24:50,839 and about an hour 739 00:24:50,840 --> 00:24:53,389 later we posted our standard 740 00:24:53,390 --> 00:24:55,519 CloudFlare customer sites are patched. 741 00:24:55,520 --> 00:24:57,379 You don't have to worry about it sort of 742 00:24:57,380 --> 00:24:59,479 post. And this is this 743 00:24:59,480 --> 00:25:01,759 was you know, it was 744 00:25:01,760 --> 00:25:03,319 it was a thing. It was it was a bug. 745 00:25:03,320 --> 00:25:05,059 And it was starting to gain some steam. 746 00:25:05,060 --> 00:25:08,239 And then about an hour after that, 747 00:25:08,240 --> 00:25:09,709 there was a tweet from Cottenham Con. 748 00:25:09,710 --> 00:25:12,049 And I think everyone can 749 00:25:12,050 --> 00:25:13,279 know what this is. 750 00:25:13,280 --> 00:25:14,749 This is that's the next site. 751 00:25:14,750 --> 00:25:17,479 This Heartbleed itself, it was branded 752 00:25:17,480 --> 00:25:19,939 and it came out to mass media. 753 00:25:19,940 --> 00:25:22,189 So this became a really 754 00:25:22,190 --> 00:25:22,669 big deal. 755 00:25:22,670 --> 00:25:25,549 Heartbleed dotcom had a logo 756 00:25:25,550 --> 00:25:27,229 hit the mainstream press Heartbleed 757 00:25:27,230 --> 00:25:28,639 virus. I don't know if you guys remember 758 00:25:28,640 --> 00:25:30,319 that, but people were saying, oh, there's 759 00:25:30,320 --> 00:25:31,909 a Heartbleed virus out there. 760 00:25:31,910 --> 00:25:34,009 And I knew I knew 761 00:25:34,010 --> 00:25:36,199 it got really bad when my my 762 00:25:36,200 --> 00:25:38,059 mother called me and said, what? 763 00:25:38,060 --> 00:25:39,060 What's going on? 764 00:25:46,750 --> 00:25:49,599 So this was kind of a big deal and, 765 00:25:49,600 --> 00:25:52,419 well, we were finished patching so 766 00:25:52,420 --> 00:25:53,619 well, we had some time to kill. 767 00:25:53,620 --> 00:25:54,620 What are we going to do 768 00:25:55,810 --> 00:25:56,859 at this point? 769 00:25:56,860 --> 00:25:58,629 There is we decided on three things. 770 00:25:58,630 --> 00:26:01,059 And one was to help 771 00:26:01,060 --> 00:26:02,949 keep this scanner that Flippo had from 772 00:26:02,950 --> 00:26:04,329 falling over. 773 00:26:04,330 --> 00:26:06,429 The second was to turn our network into a 774 00:26:06,430 --> 00:26:08,619 large honeypot to see what type 775 00:26:08,620 --> 00:26:09,969 of attacks or what type of scans are 776 00:26:09,970 --> 00:26:10,779 happening. 777 00:26:10,780 --> 00:26:13,389 And then the third was to decide 778 00:26:13,390 --> 00:26:14,559 or to figure out what we're going to do 779 00:26:14,560 --> 00:26:15,609 about our certificate's. 780 00:26:15,610 --> 00:26:17,679 We have a quite a few websites that 781 00:26:17,680 --> 00:26:20,409 use our service and many of them use SSL 782 00:26:20,410 --> 00:26:21,669 and we had about hundred thousand 783 00:26:21,670 --> 00:26:23,889 certificates and revoking them was not 784 00:26:23,890 --> 00:26:25,989 really at this point. 785 00:26:25,990 --> 00:26:27,429 The day after disclosure, it wasn't 786 00:26:27,430 --> 00:26:29,319 absolutely clear that this was something 787 00:26:29,320 --> 00:26:30,429 that you had to do. 788 00:26:30,430 --> 00:26:32,799 So first, let's talk a little bit about 789 00:26:32,800 --> 00:26:33,789 this Heartbleed scanner. 790 00:26:33,790 --> 00:26:35,709 So Filipa, who's now a CloudFlare 791 00:26:35,710 --> 00:26:38,199 engineer, wrote this server and go 792 00:26:38,200 --> 00:26:40,239 and you type in your hostname and it 793 00:26:40,240 --> 00:26:42,309 scans it for her for 794 00:26:42,310 --> 00:26:44,769 whether or not it answers Heartbleed 795 00:26:44,770 --> 00:26:46,899 heartbeat requests that are malformed. 796 00:26:46,900 --> 00:26:48,669 So these are small ones around one 797 00:26:48,670 --> 00:26:50,109 hundred bytes so that it doesn't leak 798 00:26:50,110 --> 00:26:52,599 anything beyond your standard 799 00:26:52,600 --> 00:26:53,919 frame. It shouldn't leak any any 800 00:26:53,920 --> 00:26:56,169 information. You put it on us and 801 00:26:56,170 --> 00:26:58,329 then put it behind CloudFlare and 802 00:26:58,330 --> 00:27:00,489 shout out to Kyle Eisen from our team for 803 00:27:00,490 --> 00:27:02,229 helping keep this up. 804 00:27:02,230 --> 00:27:03,999 But this this is kind of what from 805 00:27:04,000 --> 00:27:05,769 Philippos server, what it looked like 806 00:27:05,770 --> 00:27:07,839 there was up April 807 00:27:07,840 --> 00:27:09,459 8th, up to two thousand requests per 808 00:27:09,460 --> 00:27:11,529 minute. So this was a 809 00:27:11,530 --> 00:27:14,259 very highly used tool. 810 00:27:14,260 --> 00:27:16,150 And that's nothing because 811 00:27:17,170 --> 00:27:18,549 this is the next two weeks. 812 00:27:18,550 --> 00:27:20,769 Two thousand is the the bottom tick right 813 00:27:20,770 --> 00:27:23,769 there. So up up to ten, 814 00:27:23,770 --> 00:27:25,869 ten to twenty thousand a day, 815 00:27:25,870 --> 00:27:28,089 a ten to twenty thousand 816 00:27:28,090 --> 00:27:30,309 scans per minute for the next 817 00:27:30,310 --> 00:27:32,709 two weeks. And we held it up to 818 00:27:32,710 --> 00:27:34,509 two hundred million tests in the first 819 00:27:34,510 --> 00:27:35,949 two weeks. 820 00:27:35,950 --> 00:27:37,539 So with the scanner up and up and 821 00:27:37,540 --> 00:27:38,540 running. Yeah. 822 00:27:43,820 --> 00:27:45,769 And thank you to Filipa, wherever he is, 823 00:27:45,770 --> 00:27:47,989 he's somewhere in the room stand up. 824 00:27:47,990 --> 00:27:49,220 This is thanks for the tool. 825 00:27:52,380 --> 00:27:53,849 I don't I don't see I'm somewhere. 826 00:27:55,930 --> 00:27:57,559 But in any case, this is this is what 827 00:27:57,560 --> 00:27:59,829 what he found in terms 828 00:27:59,830 --> 00:28:02,019 of Domain's, it 829 00:28:02,020 --> 00:28:03,009 was really bad at first. 830 00:28:03,010 --> 00:28:04,359 This is the knife. 831 00:28:04,360 --> 00:28:06,459 This is two days after Heartbleed was 832 00:28:06,460 --> 00:28:07,719 originally announced. 833 00:28:07,720 --> 00:28:09,969 Up to 30 percent of the sites 834 00:28:09,970 --> 00:28:11,559 that he scanned were vulnerable. 835 00:28:11,560 --> 00:28:13,659 And this luckily cut down. 836 00:28:13,660 --> 00:28:15,759 And a lot of people use this in their 837 00:28:15,760 --> 00:28:18,009 automated testing to validate 838 00:28:18,010 --> 00:28:19,929 sites. But, yeah, it got down to a low, 839 00:28:19,930 --> 00:28:21,849 low number pretty pretty quickly. 840 00:28:21,850 --> 00:28:24,229 So now that that was up and running back 841 00:28:24,230 --> 00:28:26,289 at CloudFlare, we decide, 842 00:28:26,290 --> 00:28:28,089 well, what can we do? 843 00:28:28,090 --> 00:28:30,189 Well, we can log every heartbeat 844 00:28:30,190 --> 00:28:32,019 that we see that comes in with with a bad 845 00:28:32,020 --> 00:28:34,119 size. And, well, we can 846 00:28:34,120 --> 00:28:36,279 put that data on the shelf until now. 847 00:28:36,280 --> 00:28:37,779 And that's that's kind of what we did. 848 00:28:37,780 --> 00:28:39,849 So here are logs from 849 00:28:39,850 --> 00:28:42,009 the 9th and sixty 850 00:28:42,010 --> 00:28:44,379 nine percent of them had a message size 851 00:28:44,380 --> 00:28:46,779 of sixteen three, eighty four, 852 00:28:46,780 --> 00:28:49,149 which you might 853 00:28:49,150 --> 00:28:51,249 know as the largest power of two, 854 00:28:51,250 --> 00:28:53,289 you can fit into a signed sixteen bit 855 00:28:53,290 --> 00:28:54,579 integer. 856 00:28:54,580 --> 00:28:56,739 But you might also recognize it as 857 00:28:56,740 --> 00:28:59,319 the hard coded value in the SSL 858 00:28:59,320 --> 00:29:01,329 test python tool that came out the first 859 00:29:01,330 --> 00:29:03,489 day, 20 percent were one 860 00:29:03,490 --> 00:29:04,809 twenty one and that was actually from 861 00:29:04,810 --> 00:29:07,299 Flippo site and University of Michigan 862 00:29:07,300 --> 00:29:09,339 maybe where you guys scanning on April 863 00:29:09,340 --> 00:29:10,600 9th or 864 00:29:12,100 --> 00:29:13,419 in any case there are a lot of requests 865 00:29:13,420 --> 00:29:15,489 that used the 866 00:29:15,490 --> 00:29:17,589 zero length packet, which is another way 867 00:29:17,590 --> 00:29:19,659 to just check to see if your site is 868 00:29:19,660 --> 00:29:20,660 vulnerable. But 869 00:29:22,390 --> 00:29:24,459 about a week later it was around the 870 00:29:24,460 --> 00:29:26,709 same. So if there were people 871 00:29:26,710 --> 00:29:28,359 who are mass exploiting this against 872 00:29:28,360 --> 00:29:30,519 sites, they were probably 873 00:29:30,520 --> 00:29:32,589 using just the basic SSL 874 00:29:32,590 --> 00:29:34,659 test Python script and 875 00:29:34,660 --> 00:29:36,339 around 20 percent were still Philippos 876 00:29:36,340 --> 00:29:38,529 tests. So if you can, 877 00:29:38,530 --> 00:29:40,689 you can do the math here. 878 00:29:40,690 --> 00:29:42,759 It's it seems that a lot of people 879 00:29:42,760 --> 00:29:44,229 were just scanning with Philippos test. 880 00:29:44,230 --> 00:29:46,239 It wasn't a lot of mass exploitation. 881 00:29:46,240 --> 00:29:48,309 And flipping the 882 00:29:48,310 --> 00:29:50,379 numbers back, you see that one percent 883 00:29:50,380 --> 00:29:52,329 of flippo scans were actually against 884 00:29:52,330 --> 00:29:53,649 sites on CloudFlare. 885 00:29:53,650 --> 00:29:56,049 So this is what the map looks like 886 00:29:56,050 --> 00:29:57,909 for where the attacks were coming from. 887 00:29:57,910 --> 00:29:59,709 I know IP maps are not really that 888 00:29:59,710 --> 00:30:00,849 interesting. They don't tell you that 889 00:30:00,850 --> 00:30:03,309 much. But this is 890 00:30:03,310 --> 00:30:05,379 this is where it is. There's some some 891 00:30:05,380 --> 00:30:08,079 strange spikes like in Iceland, but 892 00:30:08,080 --> 00:30:09,130 don't read too much into it. 893 00:30:10,420 --> 00:30:12,609 Now, the question is, and this is 894 00:30:12,610 --> 00:30:14,079 this is what we were thinking when it 895 00:30:14,080 --> 00:30:16,239 first came out was why was it 896 00:30:16,240 --> 00:30:17,829 really so dangerous? 897 00:30:17,830 --> 00:30:20,379 Why was Heartbleed so bad? 898 00:30:20,380 --> 00:30:22,899 Well, it's a kind of a layer six 899 00:30:22,900 --> 00:30:25,119 request that doesn't necessarily 900 00:30:25,120 --> 00:30:27,249 get logged. People don't log 901 00:30:27,250 --> 00:30:29,469 parts of handshake's very often unless 902 00:30:29,470 --> 00:30:31,809 you have a specific ID's rule 903 00:30:31,810 --> 00:30:33,909 or something like that. And it's really 904 00:30:33,910 --> 00:30:36,249 bad in that. Sixty four K of server 905 00:30:36,250 --> 00:30:38,739 memory can be infiltrated 906 00:30:38,740 --> 00:30:39,999 by one request. 907 00:30:40,000 --> 00:30:42,189 And this has login, login info, session 908 00:30:42,190 --> 00:30:44,799 cookies and perhaps 909 00:30:44,800 --> 00:30:46,419 two private keys. 910 00:30:46,420 --> 00:30:47,499 We didn't know. 911 00:30:47,500 --> 00:30:49,719 And if you kind of look 912 00:30:49,720 --> 00:30:51,789 at the diagram here, this is that's the 913 00:30:51,790 --> 00:30:52,809 heap right there. 914 00:30:54,130 --> 00:30:56,469 Everything that's above the request that 915 00:30:56,470 --> 00:30:57,969 if you have a new request comes in, it 916 00:30:57,970 --> 00:30:59,919 gets put on the heap and anything that 917 00:30:59,920 --> 00:31:02,019 was previously removed from the heap 918 00:31:02,020 --> 00:31:03,279 is still sitting there. 919 00:31:03,280 --> 00:31:05,679 So we know that there are passwords, 920 00:31:05,680 --> 00:31:07,299 cookies. People were finding this right 921 00:31:07,300 --> 00:31:09,669 away. And the question 922 00:31:09,670 --> 00:31:11,679 was, would the key be there? 923 00:31:11,680 --> 00:31:13,809 What is the key going to set above one of 924 00:31:13,810 --> 00:31:14,810 these requests? 925 00:31:16,630 --> 00:31:17,799 So we looked at the code. 926 00:31:17,800 --> 00:31:20,109 Right. And what did the code 927 00:31:20,110 --> 00:31:22,359 say? Well, it said this can't happen, 928 00:31:22,360 --> 00:31:23,859 at least not in Engine X. 929 00:31:23,860 --> 00:31:25,779 The key gets loaded right away and 930 00:31:25,780 --> 00:31:26,859 therefore gets at the bottom of the 931 00:31:26,860 --> 00:31:28,959 stack. And any 932 00:31:28,960 --> 00:31:31,869 time that you do applications or 933 00:31:31,870 --> 00:31:33,429 for requests coming in, they're going to 934 00:31:33,430 --> 00:31:34,719 be higher up on the stack and they're not 935 00:31:34,720 --> 00:31:37,059 going be able to read the original key. 936 00:31:37,060 --> 00:31:38,259 Right. 937 00:31:38,260 --> 00:31:40,359 And Engine X itself was single threaded. 938 00:31:40,360 --> 00:31:42,519 So if you have this, it's not going to be 939 00:31:42,520 --> 00:31:44,649 able to catch something halfway in the 940 00:31:44,650 --> 00:31:46,869 middle of another thread doing 941 00:31:46,870 --> 00:31:47,919 an operation. 942 00:31:47,920 --> 00:31:50,109 So open SSL 943 00:31:50,110 --> 00:31:52,479 has a big number library that they use 944 00:31:52,480 --> 00:31:53,889 and they clear the memory when they're 945 00:31:53,890 --> 00:31:56,049 done. So if you're doing a handshake for 946 00:31:56,050 --> 00:31:58,539 TLS, all the cryptographic 947 00:31:58,540 --> 00:32:00,609 material is going to be cleared by open 948 00:32:00,610 --> 00:32:01,610 SSL. 949 00:32:02,290 --> 00:32:03,849 At least that's what we thought. 950 00:32:03,850 --> 00:32:06,049 Right. And we 951 00:32:06,050 --> 00:32:07,569 we weren't sure. I mean, I just looked at 952 00:32:07,570 --> 00:32:09,159 some code and what do I know? 953 00:32:09,160 --> 00:32:11,499 So we launched the CloudFlare 954 00:32:11,500 --> 00:32:13,329 Heartbleed challenge. 955 00:32:13,330 --> 00:32:15,219 So it was a this was something that we 956 00:32:15,220 --> 00:32:16,750 did to crowdsource an answer. 957 00:32:17,890 --> 00:32:19,959 So we set up a standard Engine X, 958 00:32:19,960 --> 00:32:21,669 which was outside of CloudFlare. 959 00:32:21,670 --> 00:32:24,129 It was on a third party VPs, and 960 00:32:24,130 --> 00:32:25,929 it had the vulnerable version of open 961 00:32:25,930 --> 00:32:28,209 SSL. And we said to 962 00:32:28,210 --> 00:32:30,459 all of you guys, come and find 963 00:32:30,460 --> 00:32:32,529 it, see what you can do and to 964 00:32:32,530 --> 00:32:34,599 to show proof, give us 965 00:32:34,600 --> 00:32:36,429 a message signed with that private key. 966 00:32:37,730 --> 00:32:39,229 What did we find? 967 00:32:39,230 --> 00:32:41,149 Well, for the first couple of hours, 968 00:32:41,150 --> 00:32:43,519 there was there was trolling and 969 00:32:43,520 --> 00:32:45,859 as you can see, somebody clapping 970 00:32:45,860 --> 00:32:46,789 because they did this. 971 00:32:46,790 --> 00:32:49,099 But basically 972 00:32:49,100 --> 00:32:50,899 anything that you post onto the page is 973 00:32:50,900 --> 00:32:52,549 going to be put into memory and the 974 00:32:52,550 --> 00:32:54,679 engine looks. So people were 975 00:32:54,680 --> 00:32:56,539 posting private keys in there and they 976 00:32:56,540 --> 00:32:58,849 were posting what looked like 977 00:32:58,850 --> 00:33:00,589 you can see my name there, Nick, what 978 00:33:00,590 --> 00:33:01,939 looked like a password's file. 979 00:33:01,940 --> 00:33:03,589 So everyone was getting really confused 980 00:33:03,590 --> 00:33:05,329 getting all this. There's a private key 981 00:33:05,330 --> 00:33:07,519 in my Heartbleed question, but nobody was 982 00:33:07,520 --> 00:33:09,619 actually getting the key until we saw 983 00:33:09,620 --> 00:33:12,529 this tweet from a fettah 984 00:33:12,530 --> 00:33:13,530 and. 985 00:33:15,220 --> 00:33:17,409 We took a look, this is 986 00:33:17,410 --> 00:33:19,059 our office. It's me pointing out a 987 00:33:19,060 --> 00:33:21,579 television screen and 988 00:33:21,580 --> 00:33:22,689 yeah, he solved it. 989 00:33:22,690 --> 00:33:24,130 So congrats to Fettah. 990 00:33:32,190 --> 00:33:35,219 And so he wasn't the only one. 991 00:33:35,220 --> 00:33:37,019 This was in the first twenty four hours 992 00:33:37,020 --> 00:33:38,969 or so, there were 12 people in the first 993 00:33:38,970 --> 00:33:40,829 48. About twenty five people had had 994 00:33:40,830 --> 00:33:43,199 solved it and got the real key and 995 00:33:43,200 --> 00:33:45,299 sent proof. 996 00:33:45,300 --> 00:33:47,549 So can you still 997 00:33:47,550 --> 00:33:48,539 private keys? 998 00:33:48,540 --> 00:33:49,889 Absolutely, yes. 999 00:33:49,890 --> 00:33:52,299 And it was solved in under 10 hours 1000 00:33:52,300 --> 00:33:53,969 at private keys can definitely be 1001 00:33:53,970 --> 00:33:55,619 vulnerable. But another thing we did was 1002 00:33:55,620 --> 00:33:57,749 we logged where in memory the 1003 00:33:57,750 --> 00:33:59,969 Heartbleed requests were coming in and 1004 00:33:59,970 --> 00:34:02,069 we compared that relative to where the 1005 00:34:02,070 --> 00:34:04,289 private he was initially allocated 1006 00:34:04,290 --> 00:34:06,359 and they they never overlapped. 1007 00:34:06,360 --> 00:34:08,428 So how did this how did it 1008 00:34:08,429 --> 00:34:09,479 actually get solved? 1009 00:34:10,860 --> 00:34:13,079 Well, there 1010 00:34:13,080 --> 00:34:14,519 was yeah, there was a second bug and open 1011 00:34:14,520 --> 00:34:17,249 SSL who would have known 1012 00:34:17,250 --> 00:34:18,250 looking through that code. 1013 00:34:24,199 --> 00:34:26,519 If you dump the memory of of the request, 1014 00:34:26,520 --> 00:34:28,849 all the places in red are where 1015 00:34:28,850 --> 00:34:31,039 key private keys did exist at one 1016 00:34:31,040 --> 00:34:32,869 point and it turns out some temporary 1017 00:34:32,870 --> 00:34:34,488 variables were not wiped. 1018 00:34:34,489 --> 00:34:35,959 This is the this is the code to clean up 1019 00:34:35,960 --> 00:34:38,029 the mess. There's a big name, 1020 00:34:38,030 --> 00:34:39,499 free versus big, dumb, clear, free. 1021 00:34:39,500 --> 00:34:41,779 This is just yeah. 1022 00:34:41,780 --> 00:34:43,369 It was just in certain cases in the 1023 00:34:43,370 --> 00:34:44,718 Montgomery multiplication, they didn't 1024 00:34:44,719 --> 00:34:47,299 clear up. They didn't clean the partial 1025 00:34:47,300 --> 00:34:48,300 pieces. 1026 00:34:48,960 --> 00:34:50,359 We can do a little bit of math just to 1027 00:34:50,360 --> 00:34:52,519 show how people actually 1028 00:34:52,520 --> 00:34:53,569 solved it with this. 1029 00:34:53,570 --> 00:34:55,789 But RSA, you have a 1030 00:34:55,790 --> 00:34:56,779 couple of different things. You have a 1031 00:34:56,780 --> 00:34:58,429 public exponent e 1032 00:34:59,750 --> 00:35:02,119 you have two primes multiplied 1033 00:35:02,120 --> 00:35:03,769 together. They make the public key and 1034 00:35:03,770 --> 00:35:05,839 the private exponent D and 1035 00:35:05,840 --> 00:35:07,909 any one of the if you get any one 1036 00:35:07,910 --> 00:35:10,129 of P, Q or or D, 1037 00:35:10,130 --> 00:35:11,869 you get the whole private key. 1038 00:35:11,870 --> 00:35:14,509 So what people did was they took 1039 00:35:14,510 --> 00:35:16,309 every one hundred, twenty eight bit block 1040 00:35:16,310 --> 00:35:18,479 that they saw in X 1041 00:35:18,480 --> 00:35:19,999 filtrated sharply data and they just 1042 00:35:20,000 --> 00:35:22,159 tried to divide it into the modulus. 1043 00:35:22,160 --> 00:35:24,409 And if it divided, there 1044 00:35:24,410 --> 00:35:25,879 you go, it's factored. 1045 00:35:25,880 --> 00:35:27,649 And this is how nine of the ten people 1046 00:35:27,650 --> 00:35:29,749 solved it turns out that one of the prime 1047 00:35:29,750 --> 00:35:32,359 factors is just sitting there on the heap 1048 00:35:32,360 --> 00:35:34,519 after tens of thousands of requests, 1049 00:35:34,520 --> 00:35:36,379 you might look upon it. 1050 00:35:36,380 --> 00:35:38,779 But one enterprising 1051 00:35:38,780 --> 00:35:39,919 gentleman, Rubenstein, who's 1052 00:35:40,940 --> 00:35:43,099 at University of Cambridge, he 1053 00:35:43,100 --> 00:35:45,379 used a much clever method, which is 1054 00:35:45,380 --> 00:35:47,419 coppersmiths attack, which is a Latisse 1055 00:35:47,420 --> 00:35:49,489 reduction attack, where you 1056 00:35:49,490 --> 00:35:51,589 only really need about 60 percent 1057 00:35:51,590 --> 00:35:54,049 of one of the private keys to to 1058 00:35:54,050 --> 00:35:55,789 to find it out. And this depends on the 1059 00:35:55,790 --> 00:35:57,949 fact that the public exponent 1060 00:35:57,950 --> 00:35:59,179 is small. So 1061 00:36:00,230 --> 00:36:01,789 for performance reasons, the public 1062 00:36:01,790 --> 00:36:04,219 exponent in RSA is small so that 1063 00:36:04,220 --> 00:36:06,379 any public operation is is 1064 00:36:06,380 --> 00:36:07,369 really fast. 1065 00:36:07,370 --> 00:36:09,469 But he solved it in only 50 1066 00:36:09,470 --> 00:36:11,569 requests. So this was 1067 00:36:11,570 --> 00:36:12,739 that was actually really interesting. 1068 00:36:22,150 --> 00:36:24,579 So private keys are gone, right? 1069 00:36:24,580 --> 00:36:25,660 What does that mean, 1070 00:36:27,280 --> 00:36:28,600 revocation time? 1071 00:36:31,210 --> 00:36:33,099 You know, the Internet was built for 1072 00:36:33,100 --> 00:36:34,659 this, right? I mean, people who designed 1073 00:36:34,660 --> 00:36:36,279 the peak, they said, yeah, you know, 1074 00:36:36,280 --> 00:36:37,509 people are going to revoke hundred 1075 00:36:37,510 --> 00:36:39,099 thousand certificates in twenty four 1076 00:36:39,100 --> 00:36:41,169 hours. This is just this 1077 00:36:41,170 --> 00:36:42,519 this is how we designed the system. 1078 00:36:42,520 --> 00:36:44,709 Right. And this is 1079 00:36:44,710 --> 00:36:45,879 what sounds kind of reported. 1080 00:36:45,880 --> 00:36:48,249 That's that's mostly us, actually. 1081 00:36:48,250 --> 00:36:49,329 That's that's all the Internet. 1082 00:36:49,330 --> 00:36:50,350 But that's mostly CloudFlare. 1083 00:36:51,370 --> 00:36:52,959 And this is this is what it look like. 1084 00:36:52,960 --> 00:36:54,879 The blue line is revoke certificates. 1085 00:36:54,880 --> 00:36:56,829 And as you can see, that's April 7th. 1086 00:36:56,830 --> 00:36:57,879 This is after Heartbleed. 1087 00:36:57,880 --> 00:37:00,159 This is when everybody was revoking and 1088 00:37:00,160 --> 00:37:02,919 then that green spike is is CloudFlare 1089 00:37:02,920 --> 00:37:05,109 revoking. But once we 1090 00:37:05,110 --> 00:37:07,599 revoked all these certificates, we found 1091 00:37:07,600 --> 00:37:10,029 out that, well, it didn't really 1092 00:37:10,030 --> 00:37:11,919 mean that browsers wouldn't accept these 1093 00:37:11,920 --> 00:37:13,239 certificates anymore. 1094 00:37:13,240 --> 00:37:15,489 And I can go into that 1095 00:37:15,490 --> 00:37:17,319 really quickly. There are three methods 1096 00:37:17,320 --> 00:37:19,419 to do this that were 1097 00:37:19,420 --> 00:37:21,699 built for, you know, handling 1098 00:37:21,700 --> 00:37:23,799 revocation and certificates and five or 1099 00:37:23,800 --> 00:37:26,259 nine. And first is the certificate 1100 00:37:26,260 --> 00:37:27,399 revocation list. 1101 00:37:27,400 --> 00:37:29,499 And this is just a 1102 00:37:29,500 --> 00:37:31,239 flat file with a list of certificates 1103 00:37:31,240 --> 00:37:33,339 that are revoked and 1104 00:37:33,340 --> 00:37:35,439 did as revoking one hundred thousand 1105 00:37:35,440 --> 00:37:37,599 certificates, breaks the URLs. 1106 00:37:37,600 --> 00:37:38,600 Heck, yeah, I did. 1107 00:37:39,850 --> 00:37:41,949 So the the CRL from Global 1108 00:37:41,950 --> 00:37:44,559 Sign grew from twenty two 1109 00:37:44,560 --> 00:37:45,669 to five megs. 1110 00:37:45,670 --> 00:37:46,670 Yeah. 1111 00:37:52,850 --> 00:37:54,919 This basically does the CRL 1112 00:37:54,920 --> 00:37:55,940 server and 1113 00:37:57,290 --> 00:37:59,539 lucky for this, lucky for Global on 1114 00:37:59,540 --> 00:38:01,399 CloudFlare was in front of their CRL 1115 00:38:01,400 --> 00:38:03,260 server, but unlucky for us. 1116 00:38:09,200 --> 00:38:10,999 What we're used to that kind of traffic, 1117 00:38:11,000 --> 00:38:12,859 but yeah, you can see here every three 1118 00:38:12,860 --> 00:38:15,049 hours there are waves and this had to do 1119 00:38:15,050 --> 00:38:17,149 with the cycles in which scrolls were 1120 00:38:17,150 --> 00:38:19,339 updated and Microsoft Internet Explorer 1121 00:38:19,340 --> 00:38:20,389 downloading them. 1122 00:38:20,390 --> 00:38:22,779 And that was pretty rough anyways. 1123 00:38:22,780 --> 00:38:23,699 Scrolls broken. 1124 00:38:23,700 --> 00:38:24,889 How about XP? 1125 00:38:24,890 --> 00:38:27,019 XP? Well, this is the 1126 00:38:27,020 --> 00:38:28,819 online certificate status protocol. 1127 00:38:28,820 --> 00:38:31,159 This is a kind of question answer. 1128 00:38:31,160 --> 00:38:33,289 Is this certificate revoked, yes or 1129 00:38:33,290 --> 00:38:35,779 no? And well, 1130 00:38:35,780 --> 00:38:36,829 it's really broken. 1131 00:38:36,830 --> 00:38:38,959 And Chrome has kind of known this 1132 00:38:38,960 --> 00:38:40,699 for a while and stopped checking. 1133 00:38:40,700 --> 00:38:43,249 But if you have hard fails, it does 1134 00:38:43,250 --> 00:38:45,259 disallow you from using certain networks, 1135 00:38:45,260 --> 00:38:47,149 especially with captive portals. 1136 00:38:47,150 --> 00:38:49,369 And if you have a soft fail, 1137 00:38:49,370 --> 00:38:50,989 somebody on the network can just kind of 1138 00:38:50,990 --> 00:38:53,689 drop the okapi response and voila, 1139 00:38:53,690 --> 00:38:55,219 the site is no longer revoked. 1140 00:38:55,220 --> 00:38:57,649 So it really doesn't work 1141 00:38:57,650 --> 00:38:59,839 or scale to the 1142 00:38:59,840 --> 00:39:01,129 degree that we'd want it to. 1143 00:39:01,130 --> 00:39:03,199 In plain Hoxby being requested from the 1144 00:39:03,200 --> 00:39:04,280 browser in that sense. 1145 00:39:05,360 --> 00:39:06,379 How about zero sets? 1146 00:39:06,380 --> 00:39:08,809 This is Google's proprietary method 1147 00:39:08,810 --> 00:39:11,509 that they basically collect 1148 00:39:11,510 --> 00:39:13,519 all the calls that they can from 1149 00:39:13,520 --> 00:39:15,589 different certificate authorities and 1150 00:39:15,590 --> 00:39:17,059 put them together and install them in the 1151 00:39:17,060 --> 00:39:19,069 browser. So you get updates with the 1152 00:39:19,070 --> 00:39:21,139 browser of these sets of scrolls. 1153 00:39:21,140 --> 00:39:23,719 I shouldn't say all the scrolls, but 1154 00:39:23,720 --> 00:39:25,939 it's specific certificates. 1155 00:39:25,940 --> 00:39:28,099 And this is kind of what we found out. 1156 00:39:28,100 --> 00:39:30,319 They only do EV certs and special EV 1157 00:39:30,320 --> 00:39:31,320 certs, and 1158 00:39:32,900 --> 00:39:34,669 if the browser doesn't get updated, then 1159 00:39:34,670 --> 00:39:36,139 you're not going to get an update on your 1160 00:39:36,140 --> 00:39:38,809 URL set, which is it's 1161 00:39:38,810 --> 00:39:39,979 kind of bad. 1162 00:39:39,980 --> 00:39:41,359 So CloudFlare challenge dot com. 1163 00:39:41,360 --> 00:39:43,309 Once it was solved, we revoked it and the 1164 00:39:43,310 --> 00:39:45,409 way it was, it was 1165 00:39:45,410 --> 00:39:47,449 not an easy search. But Chrome did market 1166 00:39:47,450 --> 00:39:49,249 as revoked because, well, they added it 1167 00:39:49,250 --> 00:39:50,599 manually into adjacent file. 1168 00:39:51,720 --> 00:39:54,049 So none of the hundred thousand 1169 00:39:54,050 --> 00:39:56,749 CLOUSER certificates were 1170 00:39:56,750 --> 00:40:00,049 being marked as revoked by 1171 00:40:00,050 --> 00:40:01,369 Google Chrome. 1172 00:40:01,370 --> 00:40:03,709 So we we basically 1173 00:40:03,710 --> 00:40:05,839 made a hack and I dunno if you can read 1174 00:40:05,840 --> 00:40:07,849 this, but it's the most efficient for 1175 00:40:07,850 --> 00:40:10,369 line of C++ revocation. 1176 00:40:10,370 --> 00:40:11,629 This isn't chromium. 1177 00:40:11,630 --> 00:40:13,369 It revokes all of our certificates. 1178 00:40:13,370 --> 00:40:14,329 This is a hack. 1179 00:40:14,330 --> 00:40:15,319 This is not scalable. 1180 00:40:15,320 --> 00:40:16,249 You shouldn't do it this way. 1181 00:40:16,250 --> 00:40:18,409 But it was it was how 1182 00:40:18,410 --> 00:40:19,789 it had to get done because there were no 1183 00:40:19,790 --> 00:40:21,769 valid ways of of doing revocation. 1184 00:40:23,030 --> 00:40:25,129 So, yeah, revocation is pretty 1185 00:40:25,130 --> 00:40:26,059 much broken. 1186 00:40:26,060 --> 00:40:28,849 And what can we do? 1187 00:40:28,850 --> 00:40:30,829 Well, there's shorter certificate 1188 00:40:30,830 --> 00:40:32,809 expiration periods that could help that 1189 00:40:32,810 --> 00:40:34,129 would at least help with the cartels 1190 00:40:34,130 --> 00:40:36,289 because you don't have to be holding on 1191 00:40:36,290 --> 00:40:38,509 to old certificates for very long. 1192 00:40:38,510 --> 00:40:40,429 You can sort of shrink the size of the 1193 00:40:40,430 --> 00:40:42,019 sets pretty quickly. 1194 00:40:42,020 --> 00:40:44,089 Okapi must staple is 1195 00:40:44,090 --> 00:40:46,219 an extension that requires you 1196 00:40:46,220 --> 00:40:48,479 to send the the server to send the 1197 00:40:48,480 --> 00:40:51,199 okapi response in the handshake 1198 00:40:51,200 --> 00:40:52,639 that can help to certificate 1199 00:40:52,640 --> 00:40:54,049 transparency, something else that's 1200 00:40:54,050 --> 00:40:55,969 thrown out there to solve this. 1201 00:40:55,970 --> 00:40:58,579 But none of these are 1202 00:40:58,580 --> 00:41:00,679 widespread and none of these have been 1203 00:41:00,680 --> 00:41:02,929 implemented. So something has to work 1204 00:41:02,930 --> 00:41:04,489 for revocation. 1205 00:41:04,490 --> 00:41:06,649 So I guess in summary, there are three 1206 00:41:06,650 --> 00:41:08,569 things that we did after after 1207 00:41:08,570 --> 00:41:09,570 Heartbleed. 1208 00:41:10,400 --> 00:41:12,469 We kept the scanner from falling over, 1209 00:41:12,470 --> 00:41:14,389 turned our site into a honeypot. 1210 00:41:14,390 --> 00:41:16,609 And while we definitively 1211 00:41:16,610 --> 00:41:18,979 answered that, yes, you have to revoke 1212 00:41:18,980 --> 00:41:20,750 your certificates and there's no excuse. 1213 00:41:31,970 --> 00:41:33,769 So there's a lot of takeaways from 1214 00:41:33,770 --> 00:41:34,770 Heartbleed, right? 1215 00:41:37,040 --> 00:41:39,199 I don't want to be the one to sort 1216 00:41:39,200 --> 00:41:41,029 of tell everybody how to take away what 1217 00:41:41,030 --> 00:41:43,339 to learn from this. But open source 1218 00:41:43,340 --> 00:41:44,269 disclosure is hard. 1219 00:41:44,270 --> 00:41:46,549 And this this really was the first 1220 00:41:46,550 --> 00:41:48,859 one of the year of 1221 00:41:48,860 --> 00:41:51,109 what turned out to many be many open 1222 00:41:51,110 --> 00:41:52,249 source disclosures. 1223 00:41:52,250 --> 00:41:54,349 And we did learn a lot 1224 00:41:54,350 --> 00:41:56,389 of lessons of how to do that correctly. 1225 00:41:57,590 --> 00:41:59,599 Other things that pointed out, which seem 1226 00:41:59,600 --> 00:42:01,399 obvious to people, but they weren't 1227 00:42:01,400 --> 00:42:03,499 obvious in or weren't in open SSL, 1228 00:42:03,500 --> 00:42:05,419 which is features should be disabled by 1229 00:42:05,420 --> 00:42:07,909 default. Nobody who installed open SSL 1230 00:42:07,910 --> 00:42:10,669 one oh one wanted 1231 00:42:10,670 --> 00:42:13,639 heartbeat's necessarily so 1232 00:42:13,640 --> 00:42:15,139 turn-off features by default. 1233 00:42:23,910 --> 00:42:26,219 Another thing is, while expect 1234 00:42:26,220 --> 00:42:28,559 the unexpected, that's sort of obvious 1235 00:42:28,560 --> 00:42:29,969 in computer security, that we didn't 1236 00:42:29,970 --> 00:42:31,709 really learn that from Heartbleed, but it 1237 00:42:31,710 --> 00:42:33,779 was it was definitely a shock when it 1238 00:42:33,780 --> 00:42:34,780 came 1239 00:42:35,880 --> 00:42:37,289 to other things. 1240 00:42:37,290 --> 00:42:39,509 These attacks, a lot of the attacks on 1241 00:42:39,510 --> 00:42:41,039 real sites like CloudFlare saw we saw 1242 00:42:41,040 --> 00:42:43,229 quite a few attacks as 1243 00:42:43,230 --> 00:42:44,909 as I mentioned, but a lot of them were 1244 00:42:44,910 --> 00:42:46,499 still were just scans from people just 1245 00:42:46,500 --> 00:42:48,749 trying to see if their sites were alive. 1246 00:42:48,750 --> 00:42:51,239 So that's that's a reassuring sign. 1247 00:42:51,240 --> 00:42:53,519 Crowdsourcing was effective for the 1248 00:42:53,520 --> 00:42:55,079 challenge. I couldn't find the private 1249 00:42:55,080 --> 00:42:55,409 keys. 1250 00:42:55,410 --> 00:42:57,539 And luckily, there are 1251 00:42:57,540 --> 00:42:59,449 very smart people out there who were able 1252 00:42:59,450 --> 00:43:01,679 to find it. Is anybody here in the in 1253 00:43:01,680 --> 00:43:03,809 the auditorium winner of the 1254 00:43:03,810 --> 00:43:05,369 CloudFlare challenge? 1255 00:43:06,570 --> 00:43:08,879 Is anybody here? I don't see any hands, 1256 00:43:08,880 --> 00:43:10,409 but anyways. 1257 00:43:11,460 --> 00:43:13,469 I don't have my glasses, so 1258 00:43:13,470 --> 00:43:15,569 congratulations wherever you are, 1259 00:43:15,570 --> 00:43:17,909 and the last thing is, is revocation 1260 00:43:17,910 --> 00:43:19,079 needs a solution. 1261 00:43:19,080 --> 00:43:21,599 And the last 1262 00:43:21,600 --> 00:43:24,569 conclusion from this is, is really, 1263 00:43:24,570 --> 00:43:27,089 you know, support open SSL. 1264 00:43:27,090 --> 00:43:28,829 It's it's. 1265 00:43:38,630 --> 00:43:40,159 I messed up my microphone there, but 1266 00:43:42,500 --> 00:43:43,500 thanks. 1267 00:43:45,690 --> 00:43:47,699 No, really, I mean, this is this is part 1268 00:43:47,700 --> 00:43:49,799 of critical infrastructure for 1269 00:43:49,800 --> 00:43:51,869 the world and for not only websites, 1270 00:43:51,870 --> 00:43:54,139 but as here was telling 1271 00:43:54,140 --> 00:43:56,459 embedded devices and 1272 00:43:56,460 --> 00:43:58,589 these guys need support. So please 1273 00:43:58,590 --> 00:44:00,749 support open SSL and I'm done. 1274 00:44:00,750 --> 00:44:01,750 Thank you. 1275 00:44:12,820 --> 00:44:14,529 OK, quick announcement before we start 1276 00:44:14,530 --> 00:44:16,899 the Q&A, if you are going to leave 1277 00:44:16,900 --> 00:44:19,059 the room, please get up now, go 1278 00:44:19,060 --> 00:44:21,219 out quietly without talking so we can do 1279 00:44:21,220 --> 00:44:23,469 the Q&A nice and quickly. 1280 00:44:23,470 --> 00:44:25,869 Also, you will be only able to leave 1281 00:44:25,870 --> 00:44:27,070 the room at this point. 1282 00:44:29,230 --> 00:44:30,639 Yes. If you have any questions, please 1283 00:44:30,640 --> 00:44:32,919 line up at the microphones. 1284 00:44:32,920 --> 00:44:34,989 We'll start with microphone number one. 1285 00:44:37,450 --> 00:44:38,689 Thanks. Not really a question. 1286 00:44:38,690 --> 00:44:41,049 Thanks. Thanks. HERTOG We really, really 1287 00:44:41,050 --> 00:44:43,149 followed the progress 1288 00:44:43,150 --> 00:44:45,009 even before the talk with your paper 1289 00:44:45,010 --> 00:44:47,629 published. And we just want to contribute 1290 00:44:47,630 --> 00:44:50,499 a piece of missing information. 1291 00:44:50,500 --> 00:44:52,899 So the first 1292 00:44:52,900 --> 00:44:55,239 attacks, as you call them, 24 hours 1293 00:44:55,240 --> 00:44:57,369 after the disclosure that left for 1294 00:44:57,370 --> 00:44:59,589 us, we weren't really attacking. 1295 00:44:59,590 --> 00:45:02,289 We started working on the scanner 1296 00:45:02,290 --> 00:45:04,479 about 20 minutes after the public 1297 00:45:04,480 --> 00:45:06,669 information, and it was a top 1298 00:45:06,670 --> 00:45:07,599 scanner in the region. 1299 00:45:07,600 --> 00:45:09,679 So it was a scanner and 1300 00:45:09,680 --> 00:45:11,349 used multiple different methods. 1301 00:45:11,350 --> 00:45:13,989 We just as we went, we 1302 00:45:13,990 --> 00:45:16,149 came to the conclusion that there 1303 00:45:16,150 --> 00:45:18,669 might be firewalls to be deployed 1304 00:45:18,670 --> 00:45:19,750 in between which 1305 00:45:21,370 --> 00:45:23,589 which may stop some 1306 00:45:23,590 --> 00:45:25,689 of the packets. So we use many 1307 00:45:25,690 --> 00:45:28,029 different packets in one in 1308 00:45:28,030 --> 00:45:30,099 every request. So that's also a scanner. 1309 00:45:30,100 --> 00:45:31,100 There 1310 00:45:35,020 --> 00:45:36,100 is a remote control, no. 1311 00:45:41,480 --> 00:45:43,799 A quick interruption, if you're 1312 00:45:43,800 --> 00:45:46,139 leaving on the ground 1313 00:45:46,140 --> 00:45:48,809 floor, please only use the front left 1314 00:45:48,810 --> 00:45:51,429 and the front door. 1315 00:45:51,430 --> 00:45:53,729 If you're up there, you can use any 1316 00:45:53,730 --> 00:45:55,439 door you want. But if you're down here, 1317 00:45:55,440 --> 00:45:57,569 please only use this door and 1318 00:45:57,570 --> 00:45:58,829 this door to leave the room. 1319 00:45:58,830 --> 00:45:59,830 Thank you. 1320 00:46:02,210 --> 00:46:04,539 Does my work at FRAGO. 1321 00:46:09,440 --> 00:46:10,789 Thanks for the comment. I don't think I 1322 00:46:10,790 --> 00:46:13,069 saw anything specific to you to 1323 00:46:13,070 --> 00:46:15,229 to your your account, but, yeah, it's 1324 00:46:15,230 --> 00:46:15,739 great to know. 1325 00:46:15,740 --> 00:46:17,119 So, I mean, it's one of those things is 1326 00:46:17,120 --> 00:46:18,819 really hard to tell from our perspective 1327 00:46:18,820 --> 00:46:20,149 about the attack, whether someone's 1328 00:46:20,150 --> 00:46:21,799 malicious or whether someone just 1329 00:46:21,800 --> 00:46:24,089 scanning a lot of the 1330 00:46:24,090 --> 00:46:25,549 the host that we see don't have any 1331 00:46:25,550 --> 00:46:27,799 information that identify and 1332 00:46:27,800 --> 00:46:28,909 what the intent is. 1333 00:46:31,740 --> 00:46:33,199 Microphone number two, please. 1334 00:46:33,200 --> 00:46:36,089 Hello, I have a question to CloudFlare, 1335 00:46:36,090 --> 00:46:38,279 could you stop blocking for 1336 00:46:38,280 --> 00:46:39,280 yourselves, please? 1337 00:46:50,790 --> 00:46:52,949 You make the Internet more central every 1338 00:46:52,950 --> 00:46:55,079 day, all the fancy home, but just switch 1339 00:46:55,080 --> 00:46:57,299 there and come on, the network 1340 00:46:57,300 --> 00:46:59,369 is so small you could handle live with 1341 00:46:59,370 --> 00:47:01,049 that thing is all of the bandwidth and 1342 00:47:01,050 --> 00:47:02,939 you're blocking all the notes all the 1343 00:47:02,940 --> 00:47:03,940 time. 1344 00:47:05,670 --> 00:47:08,789 We're looking into solutions for 1345 00:47:08,790 --> 00:47:11,189 the main problem is that 1346 00:47:11,190 --> 00:47:13,469 there is a lot of spam on 1347 00:47:13,470 --> 00:47:14,639 tour for Web sites. 1348 00:47:14,640 --> 00:47:16,889 And right now filtering 1349 00:47:16,890 --> 00:47:18,959 through the good and the bad is 1350 00:47:18,960 --> 00:47:21,149 something that we're working on and we 1351 00:47:21,150 --> 00:47:23,019 are focusing on that this year. 1352 00:47:23,020 --> 00:47:24,779 So look for something coming up this year 1353 00:47:24,780 --> 00:47:26,980 for Tor users like. 1354 00:47:33,740 --> 00:47:35,389 Next up, we have a question from our 1355 00:47:35,390 --> 00:47:37,699 signal Angel asked on IFC. 1356 00:47:37,700 --> 00:47:38,989 Thank you. 1357 00:47:38,990 --> 00:47:41,119 So given that maximum tearless record 1358 00:47:41,120 --> 00:47:43,399 length is 16 kilobyte, how 1359 00:47:43,400 --> 00:47:45,499 is it possible to even get 64 1360 00:47:45,500 --> 00:47:46,500 kilobytes 1361 00:47:49,020 --> 00:47:51,589 or it's 1362 00:47:51,590 --> 00:47:53,749 you can split you can split a request 1363 00:47:53,750 --> 00:47:54,800 over multiple records? 1364 00:47:55,910 --> 00:47:57,799 I believe I think I'm pretty sure that's 1365 00:47:57,800 --> 00:47:58,800 how it works. 1366 00:48:01,220 --> 00:48:03,139 Microphone number one, please. 1367 00:48:03,140 --> 00:48:04,969 What is your opinion on Libros? 1368 00:48:07,850 --> 00:48:08,850 Go for it. 1369 00:48:10,490 --> 00:48:11,490 Um, 1370 00:48:12,800 --> 00:48:14,089 I'm not sure for the most qualified 1371 00:48:14,090 --> 00:48:15,589 person to answer that. 1372 00:48:15,590 --> 00:48:17,359 To be honest. 1373 00:48:17,360 --> 00:48:19,489 I think there's well intentioned I think 1374 00:48:19,490 --> 00:48:21,409 the community acknowledges that there are 1375 00:48:21,410 --> 00:48:23,749 a lot of problems in open self 1376 00:48:23,750 --> 00:48:26,149 in terms of maintainability. 1377 00:48:26,150 --> 00:48:27,439 It's one solution. 1378 00:48:27,440 --> 00:48:28,700 I'm not sure it's the correct one. 1379 00:48:29,720 --> 00:48:31,639 Yeah, I think I think Libros itself works 1380 00:48:31,640 --> 00:48:33,739 for its goal, 1381 00:48:33,740 --> 00:48:35,389 which is OpenBSD. 1382 00:48:36,530 --> 00:48:39,109 They have a portable version by now 1383 00:48:39,110 --> 00:48:41,269 as well as they did with 1384 00:48:41,270 --> 00:48:43,729 all of our other folks. 1385 00:48:43,730 --> 00:48:46,039 And it's 1386 00:48:46,040 --> 00:48:47,059 much cleaner code. 1387 00:48:47,060 --> 00:48:49,309 And still, open SSL tries 1388 00:48:49,310 --> 00:48:51,469 to support way 1389 00:48:51,470 --> 00:48:53,689 old systems like Windows, 16 1390 00:48:53,690 --> 00:48:55,549 feet and bullshit. You really don't need 1391 00:48:55,550 --> 00:48:57,679 so down maintainability, you 1392 00:48:57,680 --> 00:48:59,190 will remain for a while. 1393 00:49:00,230 --> 00:49:02,299 Yeah, I agree that open 1394 00:49:02,300 --> 00:49:04,519 SSL does support quite a lot of 1395 00:49:04,520 --> 00:49:05,869 different platforms and some people need 1396 00:49:05,870 --> 00:49:06,769 that. 1397 00:49:06,770 --> 00:49:08,839 But the different forms of openness as 1398 00:49:08,840 --> 00:49:10,969 well that have come recently, Libras and 1399 00:49:10,970 --> 00:49:13,639 sell boring SSL are 1400 00:49:13,640 --> 00:49:15,199 taking patches from each other. 1401 00:49:15,200 --> 00:49:17,329 So I think the more people 1402 00:49:17,330 --> 00:49:19,699 looking at this project, the better. 1403 00:49:21,290 --> 00:49:22,669 We have time for one more question. 1404 00:49:22,670 --> 00:49:24,709 Microphone number three, please. 1405 00:49:24,710 --> 00:49:26,989 Hello. And both your talks, you showed 1406 00:49:26,990 --> 00:49:29,059 numbers of vulnerable sites 1407 00:49:29,060 --> 00:49:31,309 decreasing and increasing again 1408 00:49:31,310 --> 00:49:33,050 afterwards. Can you explain that? 1409 00:49:34,850 --> 00:49:37,489 So I think there are small bumps 1410 00:49:37,490 --> 00:49:39,349 that were just people coming and going. 1411 00:49:39,350 --> 00:49:42,319 I don't think we saw a lot of 1412 00:49:42,320 --> 00:49:44,029 websites that became vulnerable. 1413 00:49:44,030 --> 00:49:46,129 I think a lot of it is just pieces 1414 00:49:46,130 --> 00:49:47,599 of measurement websites that came and go 1415 00:49:47,600 --> 00:49:49,459 between different scans. 1416 00:49:49,460 --> 00:49:51,709 I don't think there are any large jumps 1417 00:49:51,710 --> 00:49:52,710 that we saw 1418 00:49:53,780 --> 00:49:55,489 when it comes to the data that I was 1419 00:49:55,490 --> 00:49:55,789 showing. 1420 00:49:55,790 --> 00:49:58,969 This is from Philippos scanner and 1421 00:49:58,970 --> 00:50:00,529 not everybody was scanning the same 1422 00:50:00,530 --> 00:50:01,999 domains every day. So that's just 1423 00:50:02,000 --> 00:50:03,079 standard variance. 1424 00:50:06,620 --> 00:50:07,669 Thank you very much. 1425 00:50:07,670 --> 00:50:09,769 Psychotronic, please give our 1426 00:50:09,770 --> 00:50:11,390 speakers a warm round of applause.