1 00:00:00,430 --> 00:00:06,600 *36C3 preroll music* 2 00:00:06,600 --> 00:00:12,730 *ominous bubbling, ebbing away* 3 00:00:19,880 --> 00:00:26,910 Herald: The next talk is called "Uncover, Understand and Own - Regaining Control 4 00:00:26,910 --> 00:00:34,260 Over Your AMD CPU", and I must say, the days where your homebrew PC would have 5 00:00:34,260 --> 00:00:40,650 been like one CPU plus a lot of discrete logic, those days are long, long gone. Now 6 00:00:40,650 --> 00:00:45,250 every single device, probably even this microphone, is full of microprocessors. 7 00:00:45,250 --> 00:00:53,490 It's pretty crazy. Robert, Alexander and Christian discovered an actual ARM 8 00:00:53,490 --> 00:01:01,390 processor on an AMD CPU, which I find quite mind boggling; and it actually 9 00:01:01,390 --> 00:01:06,540 includes its own firmware. To talk about that, I'd like to welcome them onto the 10 00:01:06,540 --> 00:01:12,579 stage. I'm really looking forward to hearing all about this discovery and what 11 00:01:12,579 --> 00:01:16,360 it has for consequences for us. So thank you very much. Give them a hand! 12 00:01:16,360 --> 00:01:22,570 *Applause* 13 00:01:22,570 --> 00:01:26,409 Robert Buhren: All right. Thanks. So, before we dive into the topic, a quick 14 00:01:26,409 --> 00:01:32,319 introduction. This is Christian and this is Alex, I'm Robert. And the reason why 15 00:01:32,319 --> 00:01:36,531 there's three of us today is, I'm a Ph.D. student at the Technische Universität in 16 00:01:36,531 --> 00:01:42,719 Berlin, and beginning of 2018, I was looking into the Secure Encrypted 17 00:01:42,719 --> 00:01:47,479 Virtualization (SEV) technology from AMD. And this technology requires a firmware 18 00:01:47,479 --> 00:01:52,640 running on the Secure Processor of AMD. And that's where Christian came into play 19 00:01:52,640 --> 00:01:57,100 because he was looking for a master's thesis. Now Christian is done with this 20 00:01:57,100 --> 00:02:03,469 thesis, and Alex here kind of took over his work. But today we're going to explain to 21 00:02:03,469 --> 00:02:09,690 you what the AMD Secure Processor is doing and what we have uncovered. So with that, 22 00:02:09,690 --> 00:02:17,100 I'm going to hand over to Christian. Christian Werling: So let's dive right 23 00:02:17,100 --> 00:02:22,010 into our first part of the presentation, which is about reverse engineering a 24 00:02:22,010 --> 00:02:27,950 completely unknown subsystem. And when we started our research, we had to find out 25 00:02:27,950 --> 00:02:33,270 what the AMD Secure Processor, formerly called Platform Security Processor, in 26 00:02:33,270 --> 00:02:38,220 this talk PSP, actually is. And it's a dedicated security subsystem that is 27 00:02:38,220 --> 00:02:46,950 integrated into your AMD CPU both on server and desktop CPUs. It's an ARM 28 00:02:46,950 --> 00:02:56,971 Cortex A5 inside your x86 CPU and it's there since around 2013. It runs a so- 29 00:02:56,971 --> 00:03:05,170 called secure OS and a kernel. And it's actually undocumented and proprietary. It 30 00:03:05,170 --> 00:03:12,340 has access to some secure off-chip storage for the firmware and some some data, and 31 00:03:12,340 --> 00:03:19,760 it mainly provides crypto functionality to the main CPU, as well as, yeah, key 32 00:03:19,760 --> 00:03:22,600 generation and key management functionality. 33 00:03:22,600 --> 00:03:29,400 It is required for the early boot. In fact, it's required for Secure Boot, and 34 00:03:29,400 --> 00:03:37,000 it acts as a trust anchor in in your system. So the PSP is a security 35 00:03:37,000 --> 00:03:44,780 subsystem, so it adds security to our system. And that's good, right? You might 36 00:03:44,780 --> 00:03:49,700 notice that this has some similarities with the Intel Management Engine, which on 37 00:03:49,700 --> 00:03:56,330 this very stage we heard a lot about three hours ago. So let's look into the 38 00:03:56,330 --> 00:04:04,380 applications of this piece of hardware. For that, we need to talk about trust. 39 00:04:04,380 --> 00:04:11,250 The one form of trust AMD tackles in what they call Secure Encrypted Virtualization (SEV). 40 00:04:11,250 --> 00:04:17,949 So you as a cloud customer can be sure that your virtual machine can even 41 00:04:17,949 --> 00:04:24,210 run in an untrusted physical location, for example, in a data center. The PSP that is 42 00:04:24,210 --> 00:04:30,759 running inside that server CPU acts as a remote, trusted entity for you as a 43 00:04:30,759 --> 00:04:38,930 customer. And it promises you to protect your memory, your data from the hypervisor 44 00:04:38,930 --> 00:04:43,620 and even from physical access. For example, through a data center 45 00:04:43,620 --> 00:04:52,930 administrator. The other form of trust that the PSP tries to establish is now 46 00:04:52,930 --> 00:05:01,880 arriving in the Linux kernel, and that's an API to a trusted execution environment. 47 00:05:01,880 --> 00:05:08,000 What that actually is, is that the PSP acts as a black box inside your system 48 00:05:08,000 --> 00:05:14,210 that is trusted by an external entity. For example, a content provider like Netflix. 49 00:05:14,210 --> 00:05:20,810 This would enable, for example, digital rights management on an untrusted system 50 00:05:20,810 --> 00:05:29,190 that is your system, like Linux. So to sum this all up, the PSP runs code that you 51 00:05:29,190 --> 00:05:35,290 don't know and that you don't control. And first of all, let's talk about the 52 00:05:35,290 --> 00:05:45,000 knowing. What you see here is a Supermicro motherboard, a server motherboard, from 53 00:05:45,000 --> 00:05:49,790 the top, and I highlighted three components here which are required or 54 00:05:49,790 --> 00:05:58,540 essential for boot up, of course. That is the CPU, the disk and so-called SPI flash. 55 00:05:58,540 --> 00:06:04,230 The SPI flash is a simple storage that is available during early boot. So if you 56 00:06:04,230 --> 00:06:09,370 look at the boot procedure in a simplified manner, then the CPU will first load the 57 00:06:09,370 --> 00:06:15,710 BIOS from this SPI flash. And only at a later stage of booting, when the necessary 58 00:06:15,710 --> 00:06:22,410 drivers are at hand, it will be able to access the hard disk to load the operating 59 00:06:22,410 --> 00:06:30,419 system. Now, as we saw from AMD's marketing slides, there is the PSP now. 60 00:06:30,419 --> 00:06:38,810 The PSP is actually part of the CPU and even boots before the CPU boots and will 61 00:06:38,810 --> 00:06:45,639 only after successful initialization of the system release the x86 CPU. So the PSP 62 00:06:45,639 --> 00:06:52,260 firmware is loaded first, and after that, the boot is proceeding as we know it with 63 00:06:52,260 --> 00:07:00,290 the BIOS and the operating system. So where is this PSP firmware coming from? 64 00:07:00,290 --> 00:07:07,340 Well, the BIOS is stored on the just- mentioned SPI flash memory and it contains 65 00:07:07,340 --> 00:07:12,850 all the data and code that is used, of course, during boot up. And it is arranged 66 00:07:12,850 --> 00:07:18,560 according to the UEFI image specification. So it's a standardized format. That's 67 00:07:18,560 --> 00:07:28,470 that's good. So maybe we should have a look into a Supermicro UEFI update. You 68 00:07:28,470 --> 00:07:34,720 see screenshots from the open source tool, UEFI tool, which is able to parse the UEFI 69 00:07:34,720 --> 00:07:40,770 image specification. You see information, for example, like the full size. This is 70 00:07:40,770 --> 00:07:44,570 16 megabytes. That's the traditional, that's the size of a traditional SPI 71 00:07:44,570 --> 00:07:53,210 flash. And you see several volumes which contain BIOS code and data. What you can 72 00:07:53,210 --> 00:07:58,510 also spot are two so-called paddings, non empty paddings. And these are called 73 00:07:58,510 --> 00:08:04,147 paddings by the tool because they are not part of the UEFI standard. 74 00:08:04,147 --> 00:08:09,060 And we're not able to parse them with the standardized information 75 00:08:09,060 --> 00:08:16,850 available. So let's use another tool. Probably many of you know "binwalk", a 76 00:08:16,850 --> 00:08:24,190 command line tool for extracting firmware from images and forensics in general. And 77 00:08:24,190 --> 00:08:30,080 let's look at the machine instructions we can find in that UEFI update for the 78 00:08:30,080 --> 00:08:37,220 Supermicro board. So the second block you see are Intel x86 instructions. This is 79 00:08:37,220 --> 00:08:44,940 what we expect, right? It's a BIOS update for an x86 CPU. So that's not surprising. 80 00:08:44,940 --> 00:08:51,570 What is more surprising are the ARM instructions. So we might be very close to 81 00:08:51,570 --> 00:09:03,430 the PSP firmware. And what we found out by staring at bytes and a hex editor a lot is 82 00:09:03,430 --> 00:09:08,540 what we call the firmware file system of the Platform Security Processor. And the 83 00:09:08,540 --> 00:09:14,230 central data structure in it is the directory. A directory starts with a 84 00:09:14,230 --> 00:09:21,420 magic string, in this case, dollar PSP, and it will have a checksum. It will have 85 00:09:21,420 --> 00:09:27,560 a number of elements that it will list and a field we don't know. And then with each 86 00:09:27,560 --> 00:09:35,420 line in the screenshot, you will have an entry in this directory. And each entry 87 00:09:35,420 --> 00:09:42,260 has a type and a size and an address where it is located inside that UEFI image. So 88 00:09:42,260 --> 00:09:47,320 the last entry of this directory is a special entry. It points to a secondary 89 00:09:47,320 --> 00:09:56,680 directory or that's how we call it. It's a continuation of this directory, and each 90 00:09:56,680 --> 00:10:02,520 entry points to something like a file. A file definitely has a body and it might 91 00:10:02,520 --> 00:10:08,260 have a header and a signature. But I'm gonna go into detail about this in just a 92 00:10:08,260 --> 00:10:13,390 second. So now we just need a reliable entry point to parse this whole firmware 93 00:10:13,390 --> 00:10:17,750 file system, and this is the Firmware Entry Table. The Firmware Entry Table 94 00:10:17,750 --> 00:10:22,500 begins with a specific byte sequence, that's how you can find it. And, it lists 95 00:10:22,500 --> 00:10:29,310 pointers to firmware blobs such as those directories inside the UEFI image. Earlier 96 00:10:29,310 --> 00:10:33,170 versions of the Firmware Entry Table are documented in source code of the Coreboot 97 00:10:33,170 --> 00:10:37,360 project, an open source BIOS implementation, and that was very helpful 98 00:10:37,360 --> 00:10:44,680 in the beginning of our research. So, to make use of all that knowledge and all 99 00:10:44,680 --> 00:10:50,620 that staring at bytes here, we developed "psptool", a command line utility that is 100 00:10:50,620 --> 00:11:01,630 able to parse any AMD firmware from UEFI updates such as the Supermicro update. And 101 00:11:01,630 --> 00:11:06,589 in the output you will see something like a directory header here, you will find 102 00:11:06,589 --> 00:11:12,170 entries like something called PSP Firmware Bootloader. You will see that it has a 103 00:11:12,170 --> 00:11:17,490 version, and psptool will even try to find out whether it's compressed, signed, 104 00:11:17,490 --> 00:11:25,480 will try to verify the signature and so on. And, just as a recap here, you can see 105 00:11:25,480 --> 00:11:29,320 that the last entry of this directory actually points to another directory, 106 00:11:29,320 --> 00:11:35,570 which psptool parses for you as well. So in order to enable you to look into the 107 00:11:35,570 --> 00:11:41,380 code that is running on your AMD CPU right now, psptool is available on GitHub and 108 00:11:41,380 --> 00:11:50,580 you can check it out today. So the PSP runs code we don't know. Well, now it's a 109 00:11:50,580 --> 00:11:56,180 matter of binary analysis to actually find out what it does. Let's talk about the 110 00:11:56,180 --> 00:12:06,080 control. Are we able to alter the firmware to run our own code? For that we had to 111 00:12:06,080 --> 00:12:13,970 play around with hardware, and more specifically we used an SPI programmer to 112 00:12:13,970 --> 00:12:21,950 flash any arbitrary UEFI image onto the SPI flash. After, for example, taking the 113 00:12:21,950 --> 00:12:27,730 original UEFI image and tinkering around with one byte or one bit we would then try 114 00:12:27,730 --> 00:12:37,410 to boot the system, and in most cases it just wouldn't boot. This was insufficient 115 00:12:37,410 --> 00:12:42,240 because we only had binary output from these experiments. So we also used the 116 00:12:42,240 --> 00:12:49,339 logic analyzer that you can see in the top of this picture. A logic analyzer is just 117 00:12:49,339 --> 00:12:54,400 an electronic instrument that can capture the data that runs through the logic 118 00:12:54,400 --> 00:13:06,029 lines. In this case, between the SPI flash and the Supermicro board. So, looking into 119 00:13:06,029 --> 00:13:16,089 a recording of one of our boot procedures we would now be able to make sense of this 120 00:13:16,089 --> 00:13:21,589 data. So, for example, we can see that the chipset here issues a read command that's 121 00:13:21,589 --> 00:13:28,240 defined by the byte three, but tried to read the address E 2 0 0 0 0 and then the 122 00:13:28,240 --> 00:13:34,720 SPI flash would gladly respond with data at that location. Now you might argue the 123 00:13:34,720 --> 00:13:38,560 data is not that interesting because that's what we control, that's what we can 124 00:13:38,560 --> 00:13:43,870 program, that's what we can look into with psptool. So what we were more curious 125 00:13:43,870 --> 00:13:50,959 about is the order and timing of the actual accesses. And to make that a bit 126 00:13:50,959 --> 00:14:00,050 more visual we wrote "psptrace". So, psptrace takes such a SPI capture and 127 00:14:00,050 --> 00:14:09,140 correlates it to the output from psptool, and we will get a enumeration of the 128 00:14:09,140 --> 00:14:15,459 specific components of the PSP during boot, and I'll get into detail about this 129 00:14:15,459 --> 00:14:21,959 also in just a second. "psptrace" is available as part of the psptool 130 00:14:21,959 --> 00:14:28,630 repository. If you're more interested about our hardware in our hardware setup, 131 00:14:28,630 --> 00:14:36,430 you can check out our talk from the CCCamp earlier this year where we actually had a 132 00:14:36,430 --> 00:14:42,970 Ryzen Pro CPU at hand, and just used the Lenovo ThinkPad. So that might be more 133 00:14:42,970 --> 00:14:52,529 suitable for your homework. So I want to share two more insights that we gained 134 00:14:52,529 --> 00:14:57,810 through our experiments in the beginning. First of all, cryptographic protections on 135 00:14:57,810 --> 00:15:05,180 files. Files are protected by signature and a field in the header determines the 136 00:15:05,180 --> 00:15:10,339 according public key that can be used to verify that signature. And that's what 137 00:15:10,339 --> 00:15:17,779 the PSP does. So there are several keys actually inside the firmware file system, 138 00:15:17,779 --> 00:15:22,579 and then all these keys are signed by the AMD root public key which does not have a 139 00:15:22,579 --> 00:15:32,240 trailing signature; but as we found out, after it is loaded from flash, it will be 140 00:15:32,240 --> 00:15:37,870 compared to a hash in Read Only Memory of the PSP. So we were not able to alter it 141 00:15:37,870 --> 00:15:47,040 like that. The second insight is how the early boot procedure of the PSP works. 142 00:15:47,040 --> 00:15:52,264 We have an on-chip bootloader that is burnt into the chip, into the PSP. 143 00:15:52,264 --> 00:15:55,431 We have an off-chip bootloader that is loaded from flash, 144 00:15:55,431 --> 00:15:59,574 and then we have several applications that are loaded subsequently. 145 00:15:59,574 --> 00:16:04,130 So now let's look a bit more closely at the output of psptrace. 146 00:16:04,130 --> 00:16:08,860 The first few read accesses are to the firmware entry table, 147 00:16:08,860 --> 00:16:15,990 the global data structure, and then the on-chip boot loader will load the PSP 148 00:16:15,990 --> 00:16:21,300 directory, it will load the AMD public key, and verify it as I just told you by 149 00:16:21,300 --> 00:16:28,720 comparing it to a hash in Read Only Memory, it will load the PSP firmware 150 00:16:28,720 --> 00:16:32,829 bootloader. That's what we called the off- chip to bootloader. And this one will be 151 00:16:32,829 --> 00:16:40,959 verified with the AMD public key. Then in the boot trace of psptrace, we see a delay 152 00:16:40,959 --> 00:16:46,300 that's due to some initialization work the PSP does, and then it will load more 153 00:16:46,300 --> 00:16:54,579 directories and will load and verify some applications eventually. And with this 154 00:16:54,579 --> 00:17:00,075 rough overview of the boot procedure, I'm gonna hand you over to Alex. 155 00:17:00,595 --> 00:17:05,895 *Applause* 156 00:17:06,420 --> 00:17:10,659 Alexander Eichner: OK. So now that we uncovered the basic modules of the 157 00:17:10,659 --> 00:17:14,002 firmware, we obviously wanted to gain deeper knowledge about what these 158 00:17:14,002 --> 00:17:18,120 individual modules do, how the firmware functions of the PSP is constructed, what 159 00:17:18,120 --> 00:17:22,959 hardware it provides and how we can interface it. So in order to do that, we 160 00:17:22,959 --> 00:17:29,370 need to do a quick recap about how AMD structures the CPU itself. So what you see 161 00:17:29,370 --> 00:17:34,330 here is a little x86 core being able to execute two threads using simultaneous 162 00:17:34,330 --> 00:17:39,080 multi threading and AMD groups four of those cores into what they call a Core 163 00:17:39,080 --> 00:17:45,000 CompleX (CCX). It contains up to four cores based on your exact model, and two 164 00:17:45,000 --> 00:17:50,330 of those complexes are put onto a CCD or Core Complex Die. That is what AMD also 165 00:17:50,330 --> 00:17:55,690 calls a chiplet. So it's a single silicon chip on your CPU and you have multiple of 166 00:17:55,690 --> 00:18:02,541 those chips on your CPU. Among the two CCXs, it contains the memory controller 167 00:18:02,541 --> 00:18:08,230 for the DDR4 memory, PCI express lanes, communication links to communicate with 168 00:18:08,230 --> 00:18:14,120 other CPUs in the system and much more. So, in our setup you saw earlier already, 169 00:18:14,120 --> 00:18:22,290 we had a two socket system with two CPUs and each of these CPUs had four CCDs. And 170 00:18:22,290 --> 00:18:29,470 now, we have not just one PSP in this whole system, but up to eight. So each of 171 00:18:29,470 --> 00:18:35,000 these CPUs or each of these little PSPs is actually executing code even before the 172 00:18:35,000 --> 00:18:43,110 x86 cores have executed anything. So AMD calls the one on CCD 0 the Master PSP, and 173 00:18:43,110 --> 00:18:48,260 all the others are referred to as slaves. The master coordinates the initial bring 174 00:18:48,260 --> 00:18:52,350 up of the platform. So for the whole initialization, for the memory controllers 175 00:18:52,350 --> 00:18:59,450 and so on and the slaves respond to requests made by the master PSP. So each 176 00:18:59,450 --> 00:19:05,081 of these PSP is identical in the system. Because they are 32 bit ARM cores, they 177 00:19:05,081 --> 00:19:10,440 have a 32 bit address space layout. The first 256KiB of this layout are backed by 178 00:19:10,440 --> 00:19:16,090 actual on-chip SRAM. The first, the on- chip bootloader, will load the off-chip 179 00:19:16,090 --> 00:19:22,200 bootloader, "PSP_FW_BOOTLOADER", and place it into memory where it will be 180 00:19:22,200 --> 00:19:27,400 executed. Among [below] the actual firmware bootloader, you will also have 181 00:19:27,400 --> 00:19:31,980 the page tables for the MMU. Yes, the PSP also has a MMU and [is] virtual-memory- 182 00:19:31,980 --> 00:19:36,760 enabled. And the code is separated into a supervisor, or kernel, mode and the user 183 00:19:36,760 --> 00:19:43,390 mode part. So, the last page you see here is the so-called boot ROM service page. It 184 00:19:43,390 --> 00:19:47,360 contains information about the PSP [that] the code is currently executing on, like 185 00:19:47,360 --> 00:19:52,500 number of sockets in the system, the current CCD ID where it's executed. It 186 00:19:52,500 --> 00:19:58,780 contains some other things, like number of sockets and so on, and it will become 187 00:19:58,780 --> 00:20:04,610 important later on. Then the off-chip bootloader will call the applications. 188 00:20:04,610 --> 00:20:08,419 They are executed in user mode. They contain the code and data to bring up the 189 00:20:08,419 --> 00:20:15,559 actual system and they also contain the stack memory. And this is done during the 190 00:20:15,559 --> 00:20:22,180 initial boot-up process by using a fixed order. And later on when the host OS runs, 191 00:20:22,180 --> 00:20:27,239 the application, for example, for the SEV functionality will be loaded on demand. 192 00:20:27,239 --> 00:20:32,809 So, the rest of the space there we have to fill is taken up by MMIO. 193 00:20:32,809 --> 00:20:36,890 So, this PSP has its own cryptographic code processor which is not shared with the 194 00:20:36,890 --> 00:20:43,221 x86. You have the hardware registers to access x86 memory, to access the system 195 00:20:43,221 --> 00:20:47,310 management network (what this is, we will come to in a bit), and much more [that] we 196 00:20:47,310 --> 00:20:53,650 don't know about now, right now. So the boot process in detail. So, Christian 197 00:20:53,650 --> 00:20:57,550 already gave you a rough overview how the boot process is done and now we will take 198 00:20:57,550 --> 00:21:01,210 a deeper look into this. So first, of course, you have the on-chip bootloader. 199 00:21:01,210 --> 00:21:05,170 It loads the off-chip bootloader from flash and executes it. The off-chip 200 00:21:05,170 --> 00:21:09,500 bootloader will execute and initialize a PSP to a bare minimum and then call the 201 00:21:09,500 --> 00:21:13,009 apps. The first one we have here, DebugUnlock and Security Gasket. We have 202 00:21:13,009 --> 00:21:16,969 no idea what they are actually for, but we named them after some strings we found in 203 00:21:16,969 --> 00:21:22,600 the binaries itself. So, the big chunk you see here is the actual bootstrapping 204 00:21:22,600 --> 00:21:27,410 phase. AMD calls it "AGESA BootLoader" (ABL) and it's not just a single binary, 205 00:21:27,410 --> 00:21:32,220 but it hosts a binary which loads binaries from the flash furthermore, and then 206 00:21:32,220 --> 00:21:37,090 executes it in a specific order. So, you see here ABL one, two, three, four and 207 00:21:37,090 --> 00:21:41,780 six. ABL five is used for something like a warm resume from suspend to RAM, for 208 00:21:41,780 --> 00:21:48,440 example. So, later on, if the SEV app is for example loaded, if the OS requests a 209 00:21:48,440 --> 00:21:55,370 specific SEV functionality and not before that. Because we have the separation 210 00:21:55,370 --> 00:21:58,950 between supervisor and user mode, we obviously need a way that the app can 211 00:21:58,950 --> 00:22:02,940 communicate with the off-chip bootloader and that is done using the ARM instruction 212 00:22:02,940 --> 00:22:09,750 "Supervisor Call" or "SVC". So we identified 76 syscalls in total. We have 213 00:22:09,750 --> 00:22:14,200 mostly reverse-engineered 30 by now. We can access the x86 memory. We can 214 00:22:14,200 --> 00:22:18,740 communicate with other PSPs in a system. We can load entries from flash and so on. 215 00:22:18,740 --> 00:22:24,400 28 are partly reverse-engineered. Those are mostly CCP operations for RSA public 216 00:22:24,400 --> 00:22:28,169 key verification, AES encryption and so on. And there are also more elaborate 217 00:22:28,169 --> 00:22:32,430 functions to communicate with other PSPs which are required during the AGESA 218 00:22:32,430 --> 00:22:37,080 BootLoader stage. And then, we have 18 left, and these we don't know about yet 219 00:22:37,080 --> 00:22:41,939 because they are not called at all or they have exactly one call site and are 220 00:22:41,939 --> 00:22:45,844 non-trivial to reverse-engineer. 221 00:22:45,844 --> 00:22:48,860 So, "System Management Network". I already saw on the 222 00:22:48,860 --> 00:22:52,580 slide already that there was access SMN. If you Google for "System Management 223 00:22:52,580 --> 00:22:57,940 Network" or SMN, you won't find much information about it by AMD or otherwise. 224 00:22:57,940 --> 00:23:02,190 The only reference you may find is code in the Linux kernel to read out the thermal 225 00:23:02,190 --> 00:23:06,750 sensors on the CPU. So the System Management Network actually is a hidden 226 00:23:06,750 --> 00:23:11,220 control network inside your CPU. Each and every hardware block which is in there is 227 00:23:11,220 --> 00:23:16,330 connected to it and is used for the PSP to control and initialize the hardware blocks 228 00:23:16,330 --> 00:23:21,300 during the boot up phase. So it is a dedicated address space, so the PSP can't 229 00:23:21,300 --> 00:23:26,630 directly access it using MMIO instructions. And we have the PSP there. 230 00:23:26,630 --> 00:23:30,820 We have identified the memory controller, the System Management Unit for which there 231 00:23:30,820 --> 00:23:35,560 was a talk about I think two years ago on this very Congress, the x86 cores are 232 00:23:35,560 --> 00:23:41,360 there as well and a lot of other things we didn't reverse engineer so far. One other 233 00:23:41,360 --> 00:23:45,910 thing. OK. So to access the System Management Network, the PSP has to map a 234 00:23:45,910 --> 00:23:49,820 certain region of the System Management Network address space into its own address 235 00:23:49,820 --> 00:23:53,929 space and then can access the register, write, read and so on. And it has to unmap 236 00:23:53,929 --> 00:23:58,370 it again. And one of the functions we identified is what we call memory 237 00:23:58,370 --> 00:24:04,620 protection slots. So the PSP has the possibility [stutters] to configure the 238 00:24:04,620 --> 00:24:09,610 memory controller, to revoke access to certain regions of the DDR4 memory from 239 00:24:09,610 --> 00:24:14,211 the x86 cores. This is done by using three registers. We have a start register with a 240 00:24:14,211 --> 00:24:18,450 physical start address, an end register to denote the physical end address of the 241 00:24:18,450 --> 00:24:22,549 region you want to protect, and a control register where we only know yet so far the 242 00:24:22,549 --> 00:24:26,620 enable bit to flip it on or off. And what it does is, if the protection is flipped 243 00:24:26,620 --> 00:24:31,110 on, the x86 will only read "all bits set"[?] when it tries to access this 244 00:24:31,110 --> 00:24:36,320 particular region and writes will have no effect through this region as well. And 245 00:24:36,320 --> 00:24:40,600 this is, for example, used for the system management mode UEFI code, and for certain 246 00:24:40,600 --> 00:24:44,795 functionality for the Secure Encrypted Virtualization feature of AMD. 247 00:24:46,965 --> 00:24:50,015 So, the next thing we did was running 248 00:24:50,015 --> 00:24:53,510 [the] `strings` [command] over all modules, obviously. And, what we found 249 00:24:53,510 --> 00:24:57,520 there were a lot of interesting debug strings and even a lot of format strings. 250 00:24:57,520 --> 00:25:02,320 And, we wanted to know what the values were during the runtime. So, when we 251 00:25:02,320 --> 00:25:06,140 disassembled the firmware and analyzed it, we saw that most of these strings were 252 00:25:06,140 --> 00:25:09,949 referenced right before a special call called SVC 6, so this must be some sort of 253 00:25:09,949 --> 00:25:16,260 debug print for the PSP. The problem is, SVC 6 is not implemented in the release 254 00:25:16,260 --> 00:25:22,210 firmware. So, we had to find another way to gain access to these debug strings. And 255 00:25:22,210 --> 00:25:28,650 this is what I will talk about now. So, the problem here is, first we need to know 256 00:25:28,650 --> 00:25:34,770 where we want to store these debug strings, and we don't have any x86 memory available 257 00:25:34,770 --> 00:25:39,030 at this time in the process. So we need to find another device or buffer where you 258 00:25:39,030 --> 00:25:44,929 can store it for later use. But, the only device we did know about at this time was 259 00:25:44,929 --> 00:25:50,669 the SPI flash. Luckily for us, right into this SPI flash area from, the PSP 260 00:25:50,669 --> 00:25:57,559 generated the necessary bus cycles on the SPI bus, without altering the flash. Then 261 00:25:57,559 --> 00:26:02,030 we need a code execution on the PSP to inject our own SVC handler. And how we 262 00:26:02,030 --> 00:26:05,850 gained code execution, Robert will talk about in the third part of this talk. But 263 00:26:05,850 --> 00:26:09,559 for now, we assume that we have code execution on the PSP already, can inject 264 00:26:09,559 --> 00:26:16,539 our own SVC 6 handler and then leave, let it run. So the app will call SVC 6, it 265 00:26:16,539 --> 00:26:20,179 will be forwarded on to the SPI bus where we can collect it with our already 266 00:26:20,179 --> 00:26:25,760 existing setup. [We] use a tool to filter the debug strings from the rest of the 267 00:26:25,760 --> 00:26:30,610 traffic on the SPI bus [that] we don't want to have [...] in the debug output and 268 00:26:30,610 --> 00:26:38,860 then hopefully get a raw PSP log. And we had success with that. So what you see 269 00:26:38,860 --> 00:26:43,789 here is the initial boot-up or the very first stage of the boot-up state. The logs 270 00:26:43,789 --> 00:26:49,170 are several megabytes long and we didn't have the chance to go through all of them. 271 00:26:49,170 --> 00:26:57,010 So, there is a lot of interesting stuff hiding there already. 272 00:26:57,010 --> 00:27:05,440 *applause* So, the next step was to explore what is 273 00:27:05,440 --> 00:27:09,510 hidden inside the System Management Network. And we didn't want to always 274 00:27:09,510 --> 00:27:13,430 reflash the whole system all the time and write code for it, debug, because that is 275 00:27:13,430 --> 00:27:20,669 error prone and tedious. So we created our own setup where we could dynamically use 276 00:27:20,669 --> 00:27:25,280 the x86 calls on the system to write and read from the System Management Network. 277 00:27:25,280 --> 00:27:29,609 For that, we replaced the SEV app with a stub and the stub provides three 278 00:27:29,609 --> 00:27:33,427 primitives. We can read-write a System Management Network address, we can execute 279 00:27:33,427 --> 00:27:37,850 an arbitrary syscall from the off-chip bootloader and we can read-write general 280 00:27:37,850 --> 00:27:45,289 PSP memory. And because the PSP is exposed as a separate PCIe device to the x86, we 281 00:27:45,289 --> 00:27:49,650 use the existing Linux kernel driver and modified it to expose these requests to 282 00:27:49,650 --> 00:27:53,850 user land, where we created a user space library wrapper and some Python bindings. 283 00:27:53,850 --> 00:27:58,970 And with that we were able to use a Python shell to dynamically read, write 284 00:27:58,970 --> 00:28:02,861 registers, headers, spurious reboot in between if you did the wrong thing, but 285 00:28:02,861 --> 00:28:06,920 could start over very quickly. So what you see here in the code snippet is, what we 286 00:28:06,920 --> 00:28:12,480 did to discover what these memory protection slots where about. You can see 287 00:28:12,480 --> 00:28:16,480 that we call an syscall handler, that we write some System Management Network 288 00:28:16,480 --> 00:28:20,890 address and so on. And we do it for all the different PSPs in the system, so the 289 00:28:20,890 --> 00:28:25,550 master PSP can also forward these requests to all of the other PSPs in the whole 290 00:28:25,550 --> 00:28:34,220 system. Next thing, we wanted to also analyze the SEV app further and see how 291 00:28:34,220 --> 00:28:39,600 the code is executed and how the data flows in this SEV app. But because we 292 00:28:39,600 --> 00:28:43,730 already had a PSP stub running there and couldn't share it on the PSP, we had to 293 00:28:43,730 --> 00:28:48,730 find another method. And we created a PSP emulator for that and using our 294 00:28:48,730 --> 00:28:55,010 libpspproxy to forward requests onto the PSP. So the current state can run the SEV 295 00:28:55,010 --> 00:29:00,890 app to a certain point and we are still actively developing that. So, that started 296 00:29:00,890 --> 00:29:08,549 a few weeks ago, and this will continue in the development. So, what it does is, what 297 00:29:08,549 --> 00:29:12,590 you see here is the AMD sev-tool to manage the host and configure all the keys and 298 00:29:12,590 --> 00:29:16,320 certificates on the system. And we modified the Linux kernel driver to 299 00:29:16,320 --> 00:29:21,039 reroute these requests out to our own PSP emulator running in user space, which is 300 00:29:21,039 --> 00:29:25,200 based on the unicorn engine. Any hardware access, because we don't know much about 301 00:29:25,200 --> 00:29:29,450 the hardware yet, is forwarded to the real PSP, results are collected, and when the 302 00:29:29,450 --> 00:29:35,080 SEV app finishes, it will return the result back to the AMD sev-tool. And with 303 00:29:35,080 --> 00:29:39,580 that we are able to execute some of the requests the SEV app implements 304 00:29:39,580 --> 00:29:46,130 successfully so far. Yeah. What you see here is a small snippet from one of the 305 00:29:46,130 --> 00:29:50,600 traces. You can see a syscall being made. It's a CCP request. We don't know 306 00:29:50,600 --> 00:29:55,450 exactly how the arguments are used by now. That's why there's a lot of unknown stuff, 307 00:29:55,450 --> 00:29:59,990 but this will aid us in development. And furthermore, in addition to allowing a 308 00:29:59,990 --> 00:30:04,070 tracing code execution and observe the data flow, we later on may be able to 309 00:30:04,070 --> 00:30:08,240 provide functionality which is currently only available on the EPYC server platform 310 00:30:08,240 --> 00:30:12,600 from AMD, like Secure Encrypted Virtual machine. The problem here is we don't know 311 00:30:12,600 --> 00:30:16,200 yet if all the hardware is there which is supported, and whether it's only a 312 00:30:16,200 --> 00:30:18,410 firmware limitation by AMD. 313 00:30:20,770 --> 00:30:24,796 If you're interested, the code is here on the repository, 314 00:30:24,796 --> 00:30:27,980 it will be made available in the next few days. We have a number of 315 00:30:27,980 --> 00:30:32,630 repositories available. You already saw PSPTool. We have some repository where we 316 00:30:32,630 --> 00:30:36,550 collect documentation about hardware interfaces, syscalls and so on. We have 317 00:30:36,550 --> 00:30:41,640 our PSP emulator there and also the psp- apps repository, if you want to dive into 318 00:30:41,640 --> 00:30:47,110 writing your own apps for the PSP. And with that I will hand over to Robert, who 319 00:30:47,110 --> 00:30:51,870 will talk about how we gained code execution on the PSP itself. 320 00:30:51,870 --> 00:30:59,108 *applause* 321 00:30:59,108 --> 00:31:02,200 Robert: OK, so for everything that Alex talked about, 322 00:31:02,200 --> 00:31:15,159 we need code execution on the PSP. … [inaudible]. Mike? Better? All right. 323 00:31:15,159 --> 00:31:22,179 So, this part of owning the PSP is again split into two parts. Now, Christian 324 00:31:22,179 --> 00:31:27,179 already talked about the firmware and the SPI flash. So, this is something we can 325 00:31:27,179 --> 00:31:31,110 control because we have physical access to the device. We can flash everything we 326 00:31:31,110 --> 00:31:36,909 want. So, what can we do with that? So, on the SPI flash, we have these directories 327 00:31:36,909 --> 00:31:41,770 which have a header and entries and an entry is actually compromised (composed) 328 00:31:41,770 --> 00:31:48,070 of an ID, an address and a size. We've talked about files. So an entry could be a 329 00:31:48,070 --> 00:31:54,419 reference to a file. And, we also talked about these secondary directories. So, an 330 00:31:54,419 --> 00:31:59,440 entry could refer to another directory. Now, if you look at the files you see that 331 00:31:59,440 --> 00:32:04,240 they have a signature usually. So, we cannot manipulate those files directly. If 332 00:32:04,240 --> 00:32:08,040 we touch them, this will be noticed and they won't be loaded and the system will 333 00:32:08,040 --> 00:32:13,620 immediately reboot. Now, what we can manipulate is the directories themselves, 334 00:32:13,620 --> 00:32:19,400 because they are not protected at all. So, specifically, what we can do is, we can, 335 00:32:19,400 --> 00:32:24,730 for example, add additional entries. These entries might point to the same files. 336 00:32:24,730 --> 00:32:29,260 That doesn't matter. We can add entries. What we also can do is, we can remove some 337 00:32:29,260 --> 00:32:35,860 of those entries or we can change entries. So, for example, this reference to the 338 00:32:35,860 --> 00:32:40,679 secondary directory, this has a size parameter. Right. And this size refers to 339 00:32:40,679 --> 00:32:44,480 the size of that directory. And actually, what we can do is, we can change that 340 00:32:44,480 --> 00:32:49,230 size. So we can make the directory appear to be smaller without removing any of 341 00:32:49,230 --> 00:32:57,140 those entries. Now, during boot, this PSP directory, that Christian already talked 342 00:32:57,140 --> 00:33:03,159 about, is parsed. So this PSP directory contains, among other things, the 343 00:33:03,159 --> 00:33:07,289 reference to the AMD public key, which is used to authenticate all the applications 344 00:33:07,289 --> 00:33:12,840 which are loaded. Now, this directory also has a secondary directory. The content is 345 00:33:12,840 --> 00:33:18,890 not really relevant here. So the on-chip bootloader that executes first will set up 346 00:33:18,890 --> 00:33:23,680 this boot ROM service page that Alex talked about. And this boot ROM service 347 00:33:23,680 --> 00:33:31,900 page contains a copy of those directory entries, just for the first directory. And 348 00:33:31,900 --> 00:33:36,270 also the on-chip bootloader will copy the AMD public key itself to the boot room 349 00:33:36,270 --> 00:33:42,529 service page. So it only copies the AMD public key if it's been verified before. 350 00:33:42,529 --> 00:33:47,580 OK. So now this boot room service page contains this AMD public key and this 351 00:33:47,580 --> 00:33:54,050 public key in memory is from then on used to authenticate applications. So the off- 352 00:33:54,050 --> 00:33:59,920 chip bootloader, which executes later, will use that boot ROM service page and 353 00:33:59,920 --> 00:34:05,070 will extend it. Specifically, it will copy the entries of the secondary directory to 354 00:34:05,070 --> 00:34:10,909 that boot ROM service page. So I guess you can already see where this is going. 355 00:34:10,909 --> 00:34:16,424 So, what could possibly go wrong here? *Laughter* 356 00:34:16,424 --> 00:34:20,700 Well, we have space for 64 entries here. And if 357 00:34:20,700 --> 00:34:26,530 we write more entries to that page, we'll hit the AMD public key. So the off-chip 358 00:34:26,530 --> 00:34:32,309 bootloader should better check that we only copy at most 64 entries. There it is. 359 00:34:32,309 --> 00:34:37,129 There is a check. Let's say this is the function that appends entries and it says: 360 00:34:37,129 --> 00:34:43,409 okay, if the number of entries exceeds 64, we return an error code and do not copy. 361 00:34:43,409 --> 00:34:48,889 Sounds good. Thing is, that number refers to the number of entries in the secondary 362 00:34:48,889 --> 00:34:56,919 directory. So this has a maximum size of 64. But there is already space, there are 363 00:34:56,919 --> 00:35:00,749 entries there on this boot ROM service page. So, actually, what we enforce with 364 00:35:00,749 --> 00:35:07,690 this check is, whatever we append can have at most 64 entries, and within that 64 365 00:35:07,690 --> 00:35:13,580 entries, well, there's the AMD public key. Super convenient. So what we do now, we 366 00:35:13,580 --> 00:35:18,309 place our own public key inside the directory structures of the firmware file 367 00:35:18,309 --> 00:35:24,579 system. The off-chip bootloader copies the entries and copies the AMD public key. 368 00:35:24,579 --> 00:35:35,369 *Applause* So what does it mean for us? Now, all this 369 00:35:35,369 --> 00:35:41,469 parsing happens before the first application is loaded. So that means we 370 00:35:41,469 --> 00:35:46,180 control the very first application and can replace the content. And from there on, we 371 00:35:46,180 --> 00:35:50,239 control the userland part of the Secure Processor. 372 00:35:51,409 --> 00:35:54,299 So, now coming to the next part. 373 00:35:54,299 --> 00:35:59,809 So, the natural next target is, of course, I mean, we have userland code execution, 374 00:35:59,809 --> 00:36:06,609 we want to have the rest. Kernel mode. So, how can we take over the kernel mode? Now, 375 00:36:06,609 --> 00:36:11,399 let's have a look at how this distinction between kernel and user mode happens. So, 376 00:36:11,399 --> 00:36:15,990 if we look at the virtual memory layout, we'll see that there is a user mode part 377 00:36:15,990 --> 00:36:21,690 and a fixed split with the kernel mode where our off-chip bootloader resides. So, 378 00:36:21,690 --> 00:36:26,560 our application, which we already control, can try to access that memory, of course, 379 00:36:26,560 --> 00:36:30,329 but that won't work. Right. The MMU will prevent any access to privileged 380 00:36:30,329 --> 00:36:39,729 memory. Okay. So let's see how this works at runtime. So, this bootloader component, 381 00:36:39,729 --> 00:36:43,819 if we specify the privileged memory a little bit more, we have code and data 382 00:36:43,819 --> 00:36:48,959 there. And at runtime another type of directory is parsed. And this is called 383 00:36:48,959 --> 00:36:53,409 the BIOS directory. I mean, it's a similar structure as the directory before. We have 384 00:36:53,409 --> 00:36:58,019 entries and the reference to a secondary directory. The entries here, again, of no 385 00:36:58,019 --> 00:37:04,900 relevance. So during boot, the off-chip bootloader will copy those entries into 386 00:37:04,900 --> 00:37:10,479 its data section. OK? So, for the copy operation, we need some some information. 387 00:37:10,479 --> 00:37:15,869 So, let's say this is the copy operation, kind of looks like `memcopy`. What we need 388 00:37:15,869 --> 00:37:21,410 is destination, where to copy? We need source. This is the secondary directory, 389 00:37:21,410 --> 00:37:26,599 this is the thing we want to copy, which is already under our control. So, 390 00:37:26,599 --> 00:37:33,160 convenient, we control whatever data is copied. And, we need a size value. So, 391 00:37:33,160 --> 00:37:39,619 where do we get that size? Oh yeah, this entry here has a size value. Super. It's 392 00:37:39,619 --> 00:37:44,720 ours also, right? We control the directory structures. We can manipulate the size. So 393 00:37:44,720 --> 00:37:49,229 to sum up, we have a copy operation into privileged memory with attacker-controlled 394 00:37:49,229 --> 00:37:56,690 data and attacker-controlled size. This is a very old meme, and I think it's 395 00:37:56,690 --> 00:38:02,170 appropriate because this this bug is so easy to prevent, actually. But for us it's 396 00:38:02,170 --> 00:38:09,819 good, because now we control everything in red here. So, we control that part. The 397 00:38:09,819 --> 00:38:16,049 thing is, as you can see, code is not part of what we control. So, what might be here? 398 00:38:16,049 --> 00:38:26,910 What is of interest for us to overwrite? Thing is, it's the page tables. The page 399 00:38:26,910 --> 00:38:31,170 tables are part of the data section within the privileged part of the virtual memory 400 00:38:31,170 --> 00:38:36,640 space. So again, what we do, we place our own page tables here. The data is copied 401 00:38:36,640 --> 00:38:41,680 and replaces the page tables in memory of the Secure Processor. So, now, if we look 402 00:38:41,680 --> 00:38:47,710 at that virtual memory overview again, well, our page tables define the virtual 403 00:38:47,710 --> 00:38:53,529 memory a bit different. We make everything user-writeable. So, we control the 404 00:38:53,529 --> 00:38:58,760 application, our application now can touch the privileged memory and just overwrite 405 00:38:58,760 --> 00:39:04,140 everything there, if we want to. For that, we need to reimplement everything. But, we 406 00:39:04,140 --> 00:39:09,869 can patch now the Secure Operating System, if we want. 407 00:39:09,869 --> 00:39:16,979 *Applause* 408 00:39:16,979 --> 00:39:19,330 So, that means, this parsing of the 409 00:39:19,330 --> 00:39:23,170 directory also happens before the first application. So, we control the first 410 00:39:23,170 --> 00:39:27,589 application, that takes over the bootloader, if you want. And from there 411 00:39:27,589 --> 00:39:34,249 on, we have everything. All those issues I presented were fixed, were even fixed 412 00:39:34,249 --> 00:39:39,430 before we discovered them. Right? So, we might not be the first one that discovered 413 00:39:39,430 --> 00:39:43,160 them. Some of you (may) remember that there was some web site called 414 00:39:43,160 --> 00:39:49,200 AMDFlaws[.com]. They did not present too many technical details. Maybe what they 415 00:39:49,200 --> 00:39:54,870 discovered was something I present here. I don't know. Thing is, it does not really 416 00:39:54,870 --> 00:39:58,340 matter for us because the Secure Processor does not implement any rollback 417 00:39:58,340 --> 00:40:03,650 prevention. So we can always go back and refresh a vulnerable firmware. And from 418 00:40:03,650 --> 00:40:12,089 that, use whatever code we want to place there. So, what what we did is, we used 419 00:40:12,089 --> 00:40:18,789 all this on an Epyc Naples based server system. And, you cannot just use that 420 00:40:18,789 --> 00:40:25,019 issue on every AMD system, because the bootloader we're using was signed with a 421 00:40:25,019 --> 00:40:31,670 key specific for the Epyc Naples CPU series. However, we believe, we have not 422 00:40:31,670 --> 00:40:35,900 tested it thoroughly yet, but we believe the same kind of issues exist in 423 00:40:35,900 --> 00:40:43,130 bootloaders which are signed with a Ryzen first generation key. And, for the rest, 424 00:40:43,130 --> 00:40:48,329 we don't know yet. So, maybe for Threadripper or Epyc Rome, there are 425 00:40:48,329 --> 00:40:54,480 similar issues, maybe not. We don't know. So the question is, is this really a 426 00:40:54,480 --> 00:41:00,190 security issue? I mean, of course it's a security issue, but, for whom? So, 427 00:41:00,190 --> 00:41:06,519 everything we did requires physical access to the device. So, if it were my laptop, 428 00:41:06,519 --> 00:41:12,910 personally, I wouldn't be concerned too much. However, there are some things where 429 00:41:12,910 --> 00:41:17,880 this is a real issue. For example, if you rely on Secure Boot. Because the Secure 430 00:41:17,880 --> 00:41:22,400 Processor is the first part that boots up, and if that is broken, everything later on 431 00:41:22,400 --> 00:41:28,391 is also broken. So, Christian already told you that AMD plans to use this Secure 432 00:41:28,391 --> 00:41:32,089 Processor [as] a trusted execution environment. If your application relies on 433 00:41:32,089 --> 00:41:38,130 that, you better not have any security issues in that Secure Processor. And, for 434 00:41:38,130 --> 00:41:43,759 the last part, the Secure Encrypted Virtualization technology from AMD is 435 00:41:43,759 --> 00:41:48,819 dependent on the integrity of the Secure Processor. If that is broken, this 436 00:41:48,819 --> 00:41:54,299 technology is also broken. So, Christian and I published a paper about that. If 437 00:41:54,299 --> 00:42:00,259 you're interested, you can read it up. But, for us here, this is actually more of 438 00:42:00,259 --> 00:42:04,859 an opportunity, right? Because we can gain more insight into this PSP with code 439 00:42:04,859 --> 00:42:10,690 execution. We can do a lot of cool things with that. So, it allows to do further 440 00:42:10,690 --> 00:42:15,950 research on other subsystems which are present in the AMD CPUs. For example, the 441 00:42:15,950 --> 00:42:22,499 PSP is responsible to load the SMU firmware. The PSP allows access to the SMM 442 00:42:22,499 --> 00:42:28,940 mode. So, this is a "ring -2 mode" on the x86 CPUs. So, [it is] higher privileged 443 00:42:28,940 --> 00:42:34,749 than your kernel, and there is proprietary code running in that mode. With the PSP, 444 00:42:34,749 --> 00:42:40,589 you have access to that code and could replace it, analyze it, whatever. And, the 445 00:42:40,589 --> 00:42:44,230 PSP is responsible to kick off the x86 calls at all. So everything that comes 446 00:42:44,230 --> 00:42:50,785 later is, in theory, now under our control. Thank you. That's it. 447 00:42:50,785 --> 00:43:00,250 *Applause* 448 00:43:00,250 --> 00:43:03,789 Herald: Yes. Thank you very much, Robert, Alexander and Christian. That was 449 00:43:03,789 --> 00:43:10,140 fantastic. Wow. I have a lot of questions I guess [in my?] head going on. But do we 450 00:43:10,140 --> 00:43:13,724 have any questions from the audience? And if you have any questions, we have 451 00:43:13,724 --> 00:43:18,380 microphones lined up here. A question is, just so that you know what we're talking 452 00:43:18,380 --> 00:43:22,930 about with questions, is a sentence with a question mark behind it and not your life 453 00:43:22,930 --> 00:43:29,190 story. And I think I saw number one first. So, let's start with number one. 454 00:43:29,190 --> 00:43:34,999 Mic 1: Hey, is there a reason why the page table is located at the end of the data 455 00:43:34,999 --> 00:43:38,279 segment? Robert: I don't think so. I mean, ... 456 00:43:38,279 --> 00:43:41,264 Mic 1: "Just because"? 457 00:43:41,264 --> 00:43:44,559 Robert: You have to place it somewhere. should be in the [interrupted] 458 00:43:44,559 --> 00:43:48,789 Mic 1: Why not in the beginning? Robert: I don't know. No idea. 459 00:43:48,789 --> 00:43:53,200 Herald: That's what I meant with "a lot of weird questions" here. From the signal 460 00:43:53,200 --> 00:43:56,229 angel we had one question. Signal Angel: And this question goes to 461 00:43:56,229 --> 00:44:02,209 the first lecturer. Didn't you have access to an SPI flasher relay(?) to attempt a 462 00:44:02,209 --> 00:44:06,599 "Time of Use versus Time of Check" [TOCTOU] attack? 463 00:44:06,599 --> 00:44:13,809 Christian: So, we had access to different tools, but the TOCTOU attack that you 464 00:44:13,809 --> 00:44:21,089 mentioned was not even necessary to mount the attacks we talked about. And actually 465 00:44:21,089 --> 00:44:26,589 so far, we don't see any possibility to mount a TOCTOU attack. 466 00:44:26,589 --> 00:44:34,029 Herald: OK. So I think I saw microphone 5 next up. Is there somebody at the 467 00:44:34,029 --> 00:44:37,039 Microphone? Mic 5: Yes. So, I was wondering if you 468 00:44:37,039 --> 00:44:40,329 considered looking at the boot ROM for issues. 469 00:44:40,329 --> 00:44:49,009 Robert: Yes, of course. The thing is, we cannot find its code in the memory any 470 00:44:49,009 --> 00:44:55,809 more after we mounted our attacks. So, I believe, the boot ROM code is not there 471 00:44:55,809 --> 00:45:03,009 anymore, which would make it much easier to analyze. We tried simple things, like 472 00:45:03,009 --> 00:45:09,239 increasing directory sizes, which are processed by the boot ROM itself. We 473 00:45:09,239 --> 00:45:12,680 haven't found any suspicious thing there, yet. 474 00:45:12,680 --> 00:45:18,469 Herald: Microphone 2. Mic 2: Thanks for your research. You have 475 00:45:18,469 --> 00:45:26,390 really nice big power over the system right now. Do you have plans to make a PSP 476 00:45:26,390 --> 00:45:35,910 firmware which is minimal and which makes your system work, but without some strange 477 00:45:35,910 --> 00:45:43,079 untrusted code? Robert: I wouldn't call it plans yet. Of 478 00:45:43,079 --> 00:45:46,839 course there are ideas to do that. The thing is, some of the functionality which 479 00:45:46,839 --> 00:45:52,020 is implemented from AMD is really required. So, the stages that Alex talked 480 00:45:52,020 --> 00:45:58,049 about, they configure and train(?) your DRAM. So without those stages, you don't 481 00:45:58,049 --> 00:46:04,839 have access to memory. Your x86 cores wouldn't work. And to reimplement that 482 00:46:04,839 --> 00:46:10,479 without having access to any manuals is really, really hard work. So, I'm not too 483 00:46:10,479 --> 00:46:13,680 confident that this will be possible in the near future. 484 00:46:13,680 --> 00:46:19,249 Mic 2: I just refer to a Management Engine cleaner, and there is such a project, 485 00:46:19,249 --> 00:46:23,760 which makes your Management Engine firmware slim. 486 00:46:23,760 --> 00:46:30,039 Robert: So, the AMD firmware is already kind of slim. The only thing that is not 487 00:46:30,039 --> 00:46:35,709 strictly required on the systems we have been looking at would be the SEV firmware, 488 00:46:35,709 --> 00:46:40,680 which is loaded on request, and you can, like, disable that by just flipping a bit 489 00:46:40,680 --> 00:46:47,190 inside that file. The system would still boot, but when it tries to initialize the 490 00:46:47,190 --> 00:46:52,180 SEV technology, the kernel would say, "OK. This does not work." The system will still 491 00:46:52,180 --> 00:46:56,109 work after that. Mic 2: Thanks. And last little question. 492 00:46:56,109 --> 00:47:03,479 Does PSP work with microcode somehow? Alexander: We didn't find anything related 493 00:47:03,479 --> 00:47:06,220 to any microcode there so far. Mic 2: Thanks. 494 00:47:06,220 --> 00:47:11,819 Herald: So let's move on to Microphone 3. Mic 3: Thank you first for the great talk. 495 00:47:11,819 --> 00:47:17,839 I have one question. Do you have maybe found something evil or potentially evil 496 00:47:17,839 --> 00:47:23,670 in the code that it does? Alexander: No. So far, they didn't find 497 00:47:23,670 --> 00:47:30,380 anything which could be used for an attack, for example. So, what the PSP 498 00:47:30,380 --> 00:47:35,940 might be able to do is, access PCIe devices. We found some code related to 499 00:47:35,940 --> 00:47:41,210 that, but we are [stutters] not sure yet whether it's actually used, because also 500 00:47:41,210 --> 00:47:46,930 the PSPs executed or is existing on graphics cards made by AMD. So, that might be also 501 00:47:46,930 --> 00:47:51,069 ready to[?] that. We couldn't find anything there yet, but so far, the PSP 502 00:47:51,069 --> 00:47:53,959 looks rather clean compared to the entire Management Engine. 503 00:47:53,959 --> 00:47:58,089 Mic 3: Thank you. Herald: So, we have a question from the 504 00:47:58,089 --> 00:48:03,119 Internet. Signal: Is the AMD public key an RSA one, 505 00:48:03,119 --> 00:48:08,680 only 576 bits? Robert: It's an RSA key, yes, but it's 506 00:48:08,680 --> 00:48:17,030 2048 bits for the first generation Epyc CPUs and I think 4069 [meaning 4096] for 507 00:48:17,030 --> 00:48:20,259 later generations. Herald: Microphone 2. 508 00:48:20,259 --> 00:48:27,469 Mic 2: For me, it seems like preventing to flash old vulnerable firmware is really 509 00:48:27,469 --> 00:48:33,069 important for a scenario like Secure Encrypted Virtualization. Can you comment 510 00:48:33,069 --> 00:48:40,430 on how difficult it is for AMD to add this retrospectively? 511 00:48:40,430 --> 00:48:47,509 Robert: Okay. So technically, rollback prevention is there for, I guess, mobile 512 00:48:47,509 --> 00:48:53,220 devices, for example. You have that. It should be possible. For adding this 513 00:48:53,220 --> 00:48:57,439 functionality afterwards, I don't think that's really possible, because the on- 514 00:48:57,439 --> 00:49:02,670 chip bootloader is the thing that loads the off-chip bootloader and verifies it. 515 00:49:02,670 --> 00:49:09,079 And that software component has to, like, stop loading if the firmware version does 516 00:49:09,079 --> 00:49:14,069 not match, for example. And you have to change that. And that functionality is not 517 00:49:14,069 --> 00:49:18,539 there and you cannot update the on-chip boot ROM. So, in that sense, I don't think 518 00:49:18,539 --> 00:49:24,089 that that's possible to change. And if you look at our paper, you will see that the 519 00:49:24,089 --> 00:49:29,289 former issues are kind of devastating for the SEV technology, because there are some 520 00:49:29,289 --> 00:49:36,209 keys which are now accessible, which can be used for attacking SEV-protected 521 00:49:36,209 --> 00:49:38,920 guests. Mic 2: Thanks. 522 00:49:38,920 --> 00:49:45,380 Herald: Microphone 3, please. Mic 3: One question. Did you analyze the 523 00:49:45,380 --> 00:49:54,640 API to the x86 core? Did you find anything that could be exploited without flashing 524 00:49:54,640 --> 00:49:59,690 anything so that you could directly go from x86 to PSP exploitation? 525 00:49:59,690 --> 00:50:06,599 Alexander: Yeah, we tried to find the necessary code to interface with the x86. 526 00:50:06,599 --> 00:50:12,179 We think we found one place where the x86 cores are released after the PSP 527 00:50:12,179 --> 00:50:15,900 initialized the whole system. But obviously, we can't do much with it except 528 00:50:15,900 --> 00:50:21,779 preventing the x86 to boot at all. And otherwise we couldn't find anything there 529 00:50:21,779 --> 00:50:26,699 yet. So we focused on, on a bit of other, like the memory controller, and didn't 530 00:50:26,699 --> 00:50:32,869 have a deeper look at the x86 interface. So what there is, the BIOS can interface 531 00:50:32,869 --> 00:50:38,719 with the PSP using a special mailbox register which is mapped in MMIO space in 532 00:50:38,719 --> 00:50:44,139 x86 for requests. So, it can, for example, the UEFI init boots, it will say to the 533 00:50:44,139 --> 00:50:48,029 PSP "Hey, this is my system management mode code region, please protect that for 534 00:50:48,029 --> 00:50:52,910 me" and it will execute this request. But apart from that, we couldn't find anything 535 00:50:52,910 --> 00:50:55,039 so far. Mic 3: Thank you. 536 00:50:55,039 --> 00:51:00,200 Herald: So, Microphone 4. Mic 4: Hi. So, is it correct that your 537 00:51:00,200 --> 00:51:08,651 work enables 100% open source firmware for this kind of processors? And if so, have 538 00:51:08,651 --> 00:51:12,979 you already contacted the CoreBoot team to make that actually happen? 539 00:51:12,979 --> 00:51:21,849 Robert: So. 100 percent open source. As for the PSP, there is this on-chip boot 540 00:51:21,849 --> 00:51:27,240 ROM which we can't replace, right? So, this will be closed source. Then there is 541 00:51:27,240 --> 00:51:33,099 code of the off-chip bootloader, until the first exploit, which runs, which is not 542 00:51:33,099 --> 00:51:38,719 open source. In theory, you could from now on take over the PSP, write your own code. 543 00:51:38,719 --> 00:51:44,150 But, as I said before, you have to reimplement a lot of functionality without 544 00:51:44,150 --> 00:51:49,369 having any documentation, right? So, technically it's possible, I guess, to do 545 00:51:49,369 --> 00:51:54,089 something like that. Practically, I'm not too sure. 546 00:51:54,089 --> 00:51:57,390 Herald: So we're gonna go to the internet for another question. 547 00:51:57,390 --> 00:52:03,039 Signal: Is it possible to block PSP from within Linux or BSD, for the system's 548 00:52:03,039 --> 00:52:09,130 runtime, by using search and boot flags? Robert: Sorry, to block what? 549 00:52:09,130 --> 00:52:14,630 Signal: To block the PSP from the Linux or BSD. 550 00:52:14,630 --> 00:52:19,680 Alexander: So, what you can do is, like Robert mentiond already, you can flip a 551 00:52:19,680 --> 00:52:25,189 bit in the SPI flash and then the PSP, once it initialized the whole system, it 552 00:52:25,189 --> 00:52:29,719 won't run the SEV app, for example, because the signatures won't match 553 00:52:29,719 --> 00:52:36,079 anymore. And there is no other sort of interface where the PSP is actually 554 00:52:36,079 --> 00:52:41,499 triggered. Or we couldn't find it so far. Herald: Microphone 3. 555 00:52:41,499 --> 00:52:45,500 Someone: I think he was first. Herald: Oh, okay, all right. Right. 556 00:52:45,500 --> 00:52:49,099 Microphone 2 then. Sorry. Mic 2: Did you try to enable any 557 00:52:49,099 --> 00:52:56,420 superpowers from PSP like JTAG or special tricks with voltage or something else? 558 00:52:56,420 --> 00:53:01,579 Robert: When the first application that is loaded has some strings in it like Debug 559 00:53:01,579 --> 00:53:08,319 Unlock. Sounds interesting. But then again, JTAG, where would you access the 560 00:53:08,319 --> 00:53:13,190 JTAG of the PSP? You need to have some connection to the lines, right? 561 00:53:13,190 --> 00:53:18,130 Mic 2: Intel supports USB debugging. Robert: Yeah, I know. With special 562 00:53:18,130 --> 00:53:21,209 devices, right? Mic 2: No, even wire cable. 563 00:53:21,209 --> 00:53:26,589 Robert: Okay. So anyhow, I have the suspicion that this DebugUnlock app is 564 00:53:26,589 --> 00:53:32,069 responsible to to allow some debug mode. Which then, I assume, with special 565 00:53:32,069 --> 00:53:36,880 hardware, you can have JTAG. But we have not touched it yet. 566 00:53:36,880 --> 00:53:39,469 Mic 3: Thanks. Herald: Now Microphone 3 567 00:53:39,469 --> 00:53:45,619 Mic 3: So I'm as far from a liar *laughing*, um, a lawyer as possible, but 568 00:53:45,619 --> 00:53:51,650 could AMD in any way file a cease and desist for anything you do? 569 00:53:51,650 --> 00:53:57,089 Robert: Probably not, I guess. Mic 3: Just curious. 570 00:53:57,089 --> 00:54:01,069 Robert: I have no idea. Mic 3: Thank you. 571 00:54:01,069 --> 00:54:06,920 Robert: And, as I said before, we're not the ones that initially discovered, or 572 00:54:06,920 --> 00:54:10,450 probably not the ones that initially discovered these issues. And it's not 573 00:54:10,450 --> 00:54:15,680 really about these issues. I mean, for me personally, these issues are a nice way to 574 00:54:15,680 --> 00:54:21,140 get more insight into the PSP. And it's not about having the super new security 575 00:54:21,140 --> 00:54:26,319 issue, whatever. So if AMD wants to file something, I guess they would have also 576 00:54:26,319 --> 00:54:32,619 filed other people that did similar research before. Maybe they did. I don't 577 00:54:32,619 --> 00:54:34,849 know. Herald: So we had another question from 578 00:54:34,849 --> 00:54:37,619 the Internet. Signal: How long did it take you to 579 00:54:37,619 --> 00:54:43,390 reverse engineer and develop all this stuff? 580 00:54:43,390 --> 00:54:51,730 Robert: So I think beginning of 2018, Christian was starting with his master's 581 00:54:51,730 --> 00:54:57,640 thesis. And we spent a lot of time on figuring out how this firmware file system 582 00:54:57,640 --> 00:55:03,579 works, and the boot process and writing these PSPTrace and PSPtool to better 583 00:55:03,579 --> 00:55:11,950 understand the components of the firmware. And Alex joined in May, May-ish this year. 584 00:55:11,950 --> 00:55:19,049 And, well, we're still working on it, right? So the emulator, once we figured 585 00:55:19,049 --> 00:55:26,439 out a lot of information about the PSP, I think the emulator was easy to develop, 586 00:55:26,439 --> 00:55:30,979 in the sense that it didn't take too much time. But of course, there was a lot 587 00:55:30,979 --> 00:55:37,569 of work going into it before that. Herald: So I do not see, oh yes I do see 588 00:55:37,569 --> 00:55:40,359 another question from the internet. Let's go for that. 589 00:55:40,359 --> 00:55:44,729 Signal: Yeah, last question. Did you try to glitch the PSP by manipulating the 590 00:55:44,729 --> 00:55:53,829 voltage of the socket(?)? Robert: Why? I think our approach is 591 00:55:53,829 --> 00:55:59,900 easier, but no, seriously, we did not try. Herald: So with that, I don't see any 592 00:55:59,900 --> 00:56:06,439 further questions. And I would like you to help me thank Robert, Alexander and 593 00:56:06,439 --> 00:56:08,161 Christian for this fantastic talk. 594 00:56:08,161 --> 00:56:09,884 *postroll music* 595 00:56:09,884 --> 00:56:21,460 Subtitles created by c3subtitles.de in the year 2020. Join, and help us!