*33C3 preroll music*
Herald-Angel: Tim ‘mithro’ Ansell has come all the way from Australia
to talk to us about ‘Dissecting HDMI’ and developing an open FPGA-based
capture hardware for sharing talks outside of the room.
And he will be explaining how to dissect it. And I’m looking forward
to hearing the talk in a second. And so please give Tim
a warm round of applause again. Thank you!
*applause*
Tim Ansell: Okay, hi, I’m Tim, and in theory, if my slides change
you would see that.
And I kind of have too many projects. And I’m gonna be discussing one of them.
This is another project that I gave a lightning talk on earlier.
If you didn’t see it, it’s an ARM microcontroller that goes in your USB port.
People wanted to know when to hack on it.
Tomorrow at 2 PM, apparently.
So, the first that I want to say is I’m a software developer.
I’m not a hardware designer, I’m not an FPGA developer,
I’m not a professional in any of that.
I develop software for full time. So this is my hobby.
As well, this information comes from a couple of projects that I started
but a lot of other people did the majority of the work and I’m just telling you
about it because they are too shy to come up and talk about it themselves.
So a big thank you to all these people who have helped me in various ways
regarding this. And theses slides –
any of the blue things are links, so if your playing it along at home
you can get to them by that URL and click on these things.
And there is probably other people I’ve forgotten, who are not on this list.
I’m very sorry.
So this title of this talk could be called “Software guy tries hardware and complains”.
I’ve had a really hard time figuring out what to call this talk.
And you’ll see some other attempts of naming this talk better.
So a bit of history. How do I end up doing HDMI stuff?
So TimVideos is a group of projects which are trying to make it easy
to record and live-stream user groups and conferences like this event.
However, we want to do it without needing the awesome team
that is doing the recording here.
These guys are really, really organized and professional.
We want to do it where people have no experience at all with AV,
and just can make it happen.
And so this is how you record a conference or user group.
I’m gonna be talking about these two things here,
the HDMI2USB devices that we created.
They are used in our setup both for camera capture and for capture of slides.
And so HDMI2USB is FOSS hardware for doing HDMI capture
and actually has a bit of history with the CCC
because it was inspired by a speaker who spoke here.
Bunnie spoke on his NeTV board
which was an FPGA Man-in-the-Middle attack
on HDCP-secured links. His talk is really awesome. It’s gonna be…
that talk is way more technical than mine and gives you
some really awesome details about the cool things he did to make that work.
Mine is much more basic. You don’t need much experience with HDMI
to follow my talk. Our device works like his does,
except his was deliberately designed to not allow capture,
our design allows capture. It effectively man-in-the-middles
the presenters projector, between the presenters laptop
and the projector, and provides a high-quality capture out the USB2 port.
I use an FPGA to do that. This is because
using FPGA makes hardware problems software problems, and as I said
I’m a software developer, I prefer software problems to hardware problems.
And the way it kind of works is it appears as a UVC webcam so that
you can use it with Skype or Hangouts or any of those things without needing
any drivers on sensible operating systems like MacOS and Linux.
On Windows you need a driver that tells it to use the internal driver.
It’s kind of weird. And also a serial port
because we have the ability to switch which input goes to which output.
It’s kind of like a matrix.
And so this the open source hardware we designed.
It’s in KiCAD, you can find it on Github.
I’m quite proud of it. It’s quite a good little kit.
We don’t use all the features of it yet, but it’s pretty awesome.
And it’s in use!
We used this technology to capture at a bunch of conferences:
PyCon in Australia, linux.conf.au in Australia
– as I said, I’m Australian.
DebConf, though, are not Australian.
They used it in – sorry – in South Africa, I think.
And there are a whole bunch of other people around the world
who are using this, which is pretty awesome.
The main reason I wanted it to be open source was so that
other people could use them and learn from them and fix problems,
because there are a lot of problems we’ve run into.
And the other thing is this is all full of Python.
We do use Python
to create the firmware for the FPGA, and all these other areas.
If you wanna find out more about that go to my talk at Python AU
which was recorded with the very device I’m talking about.
*microphone noise* Oops, sorry!
But as I said, this is going to include a lot of problems.
The first one is: People still use VGA!
This kind of makes me sad.
Because VGA is not HDMI. It was invented in 1987
and it’s an analog signal.
Well, HDMI shares some history with VGA.
You can’t use the same techniques for capturing HDMI that you can for VGA.
So why do you still use it? It’s old and bad!
We developed a VGA expansion board to effectively allow us to capture
VGA using the same thing.
By ‘developed’ I mean we have designed, and some exist
but nobody’s actually finished the firmware to make them work yet.
So, I’d love help there.
There is also another problem:
I want to do this all open source, as I said.
The HDMI ecosystem has commercial cores you can buy
and they work reasonably well, but you have to buy them
and you don’t get the source code to them or if you do get the source code to them
you can’t share them with other people.
As well, I wanted to be open source because we wanted to solve
all those problems that people have when plugging in their laptop
and it not working.
And the commercial cores aren’t designed to allow us to give the ability
to do that – solve those problems permanently.
So we create a new implementation!
As anybody who has ever done a reimplementation or a new implementation
or something, it means that you got new bugs which I all describe quite a bit.
So this talk could be called
‘Debugging HDMI’ rather than ‘Dissecting HDMI’ because it includes
a lot of information about how things went wrong.
Ok, that’s kind of the introduction of why we are here and why I’m talking about this.
So how does HDMI work?
Well, HDMI is actually reasonably old now.
It was created in 2002.
It’s based on the DVI specification. DVI was created in 1999.
So DVI is 17 years old.
And DVI was designed to replace VGA and shares a lot of similar history.
HDMI is backwards compatible with DVI electrically and protocol wise,
but uses a different connector.
This is a HDMI connector. You’ve probably seen them all before.
If you look closely you see that there are 19 pins on the HDMI connector.
That’s Pin 1.
So what do all this pins do?
There are five pins which are used for Ground.
There is one pin which is used for Power, it gives you 5 volts at 50 milliamps.
This isn’t much. You can’t do much with 50 milliamps
except maybe some type of adapter, converter or power a whole microcontroller.
Some Chinese devices try to draw like an amp from this.
That’s not very good. So that’s another thing you should watch out for.
There are three high speed data pairs which transmit the actual video data.
And they share a clock pair. So that’s these pins here.
And then there are five pins which are used for low speed data.
So that’s all the pins on the HDMI connector.
You might have noticed that there was a whole bunch of different things
I said there. And you need to actually understand a whole bunch
of different protocols to understand how HDMI works.
There is bunch of low speed ones and there is a bunch of high speed ones.
I’m not gonna talk about all of those protocols
because there is just too many to go into an hour talk.
The low speed protocol I’m not gonna talk about is
CEC or Audio Return; and I’m not gonna talk about any of the
Auxiliary Data protocol that is high speed, or HDCP.
If you want HDCP go and look at bunnie’s talk.
It’s much better than mine. But… or ethernet!
What I will be talking about is the EDID and DDC protocols,
The 8b/10b encoding of the pixel data
and the 2b/10b encoding of the control data.
Interesting enough this is actually DVI.
I’m not telling you about HDMI, I’m really describing to you how DVI works.
Again, many titles.
Starting with the low speed protocol:
EDID or DDC.
I’m gonna use these two terms interchangeably,
they’ve been so confused now that they are interchangeable,
in my opinion.
This is something they inherited from VGA.
It was invented and added to VGA in August 1994.
It was for plug and play of monitors so that you could plug in your monitor
and your graphics card would just work rather than requiring you to tell
your graphics card exactly what resolution and stuff your monitor worked at.
It uses I2C [I squared C] and a small EEPROM.
These are the pins that it uses.
15 is the Clock pin and 16 is the Data pin,
and then it uses the Ground, and the 5 Volts is used to power that EEPROM.
And in some ways it also uses the 19 because 19 is how you detect
that there is something there to read from.
It uses I2C.
I2C is a low speed protocol that runs at either 100 kHz or 400 kHz.
Technically EDID is not I2C, but it is.
It only supports the 100 kHz version, though, in theory,
everything on this planet can be read at 400 kHz.
It is also very well explained elsewhere, so I’m not going to explain in detail
what I2C is or does, or how to implement it.
The EEPROM is a 24 series. It’s found at I2C address 50.
It’s 8 bits in size which gives you 256 bytes of data.
Again, this EEPROM and how to talk to it is very well described on internet.
So I’m not gonna to describe it here. If you’ve used EEPROMs
over I2C it’s likely you’ve used a 24 series EEPROM.
Probably bigger ones, 256 bytes is pretty small.
So like a 16 bits one,
but EDID supports only the 8 bits ones.
The kind of interesting part of EDID is the data structure:
it’s a custom binary format that describes
what the contents of the EEPROM is.
Again, Wikipedia has a really good description of this.
So I’m not gonna go into much detail.
But the important thing is that it describes resolution, frequency
and format for talking to the monitor.
This is really important because if you try and send
the wrong resolution, frequency or format the monitor is not gonna understand it.
This is kind of what EDID is used for.
*sipping, sound of water bottle*
So this is where things start getting a bit hairy.
Presenters come up to the front and the first question you see anybody ask is:
What resolution do I use?
And they get a panel like this which has a bazillion of resolutions selected.
And the thing is, despite your monitor saying that it supports
many formats they lie.
It turns out that projectors lie a lot more than normal displays.
I don’t know why they are special.
So this is what a supported format looks like.
It’s really great.
As well, I care about capturing the data.
And so I want the things in the format that is
easy for me to capture.
I also don’t to be scaling peoples images and text
because scaling looks really bad. So if someone selects
like a really low resolution and we scale it up it looks really horrible.
It makes text unreadable; and presenters are very denounced,
especially at technical conferences, for using tiny, tiny fonts.
And so we need to use as much resolution as we can.
How we solve this is we emulate our own EEPROM in the FPGA
and ignore what the projector tells us it can do.
We tell the presenter that this is what we support.
You might notice that it kind of solves the problem
of what resolution we do.
Offer a single solution…
offer a single option makes it very hard to choose the wrong one.
That’s good! We solved the problem!
No, we haven’t solved the problem.
We were recording PyCon AU and we found that
some Mac laptops were refusing to work.
To understand the cause of this you need to understand
a little bit about how the world works.
There are two major frequencies in the world: 50 Hz and 60 Hz.
50 Hz is mainly used in the “Rest of the World”
and 60 Hz is used in America and Japan and a few other places
but that’s kind of a very rough thing.
Laptop sold in Australia, Australia is 50 Hz.
It’s part of the “Rest of the World”. You’d think the that the laptop
could do 50 Hz. Plus everything is global these days, right?
I can plug in my power pack for my laptop in the US or Australia,
like, it should work everywhere right!
No. Sad!
We solved it by claiming that we were American
and supporting 60 frames per second rather than 50 frames per second.
So I guess a display with an American accent.
We deployed this hotfix on the Friday evening.
And on Saturday all the problems that we were having on Friday
went away. So this is kind of the power of a open source solution
and having complete control over your hardware.
Nowadays we actually offer both 60 and 50
because for display capture if you’re displaying stuff
at 50 frames per second you’re probably speaking a lot faster than I am.
It’s really weird, these 128 bytes are really hard
and the number one cause of why a persons laptop
can’t talk to the projector.
It gets a trophy!
To try and figure out why that is we created EDID.tv.
It’s supposed to be a repository of EDID data.
There is a Summer of Code project, Python/Django/Bootstrap
and an EDID grabber tool that you can run on your laptop.
I’d love help making this work better.
Hasn’t had much love since the Summer of Code student made that work.
But it would be really nice to have an open database of everybody’s EDID data
out there. There are a bunch of closed ones. I can pay to buy one,
but I’d really love to have an open one.
As well maybe we don’t need the whole capture solution,
maybe you can just override the EDID.
The C3VOC here actually developed a version that overrides EDID for VGA.
I have a design which works for HDMI.
It just uses a low cost microprocessor to pretend to be an EEPROM.
As well, DisplayPort is not HDMI. Don’t get these two confused,
they are very, very different protocols.
They have an Auxiliary Channel like EDID and CEC.
I have boards to decode them here at CCC.
So if you’re interested in that come and talk to me
because we would really like to do similar things for DisplayPort.
That is the slow speed data.
*Sip from bottle*
What about the high speed data?
Each pixel on your screen is
basically three colors in DVI standard: Red, Green, Blue.
And each one is a byte in size.
Each of the colors is mapped to a channel on the HDMI connector.
You can kind of see the Red and the Green and the Blue channels.
Each channel is differential pair.
You get a Plus and a Negative and a Shield.
And they use twisted pair to try and reduce the noise reception of these,
because these are quite high speed.
And they have a dedicated Shield to try and – again – reduce the noise
that is captured.
This is kind of where it gets to the ‘differential signaling’ part
of the ‘TMDS’ that is the kind of code name
for the internal protocol that is used on the high speed data.
They also… all these channels share a Clock.
That clock is called the Pixel Clock.
But each of these channels is a serial channel.
It transmits data at 10 bits.
They… every 10 bits – sorry,
every clock cycle there are 10 bits of data transmitted on each of these channels.
There is a shared clock and each of the channels is running
at effectively ten times that shared clock.
This is kind of what the whole system looks like.
You have your Red, Green, Blue channels.
You take your 8 bits of input data on each channel
and you convert it to the 10 bits
that we’re going to transmit, and it goes across the cable
and then we decode it on the other side.
The question is: what does the 8 bit to 10 bit encoding
look like and how do you understand that.
It is described by this diagram here. It’s a bit small so I’ll bring it up.
This is what it looks like. Yes… sure…
…what? This diagram – like –
I’ve spent hours looking at this, and it is an extremely hard diagram
to decode. It’s very, very hard to understand.
And it turns out the encoding protocol is actually quite easy!
It’s three easy steps – approximately.
So I’m going to show you all how to write an encoder or a decoder.
That diagram is just for the encoder.
They have a similar diagram that is not the inverse of this for decoding.
Again, almost impossible to read.
The three steps: First we’re going to do ‘Control’ or ‘Pixel’,
choose which one to do. Then we’re going to either encode Control data
or encode Pixel data.
A couple of important points to go through first:
The Input data – no matter how wide it is –
is converted to 10 bit symbols.
Data goes to symbols. When we’re talking about them
being transmitted we talk about them in symbols, when it’s decoded into pixels
we talk about them in data.
As well, things need to be kept DC-balanced.
I’ve rushed ahead.
The question is: “Why 10 bits?” Our pixels were 8 bits.
I will explain why in the Pixel Data section.
But it’s important that all our symbols are the same size.
We’re always transmitting 10 bits every clock cycle.
Keeping DC-balanced:
long runs of 1s and 0s are bad.
There are lots of reasons for this.
I tend to think of it like HDMI isn’t AC coupled
but you can kind of think of it like AC coupled.
It’s not to recover Clock.
We have a clock pair that is used to give our Clock signal.
There are lots of lies on internet that say that the reason
we’re going to keep DC balance is because of Clock.
But no, that’s not the case.
So what does DC balance mean?
A symbol which has lots of 1s or lots of 0s
is going to be considered DC-biased
if it has more 1s than 0s.
This is kind of what it’s like: this symbol here
has lots of 1s and if you add up all the 1s you can see it’s got
quite a positive bias. If it was inverse and had lots of 0s
it would have a negative DC bias.
That cause… that DC bias over time causes us problems.
That are the two important things we have to keep in mind when looking at the rest.
*sound of bottle sip*
The first thing we need to figure out is are we transmitting Control data
or Pixel data.
Turns out that what is happening in your display is,
we are transmitting something that’s actually bigger
than what you see on your screen.
This not the scale. The Control data periods are much, much smaller.
The Control data is in orange and the Pixel data is in purple-pink.
So why does this exist? It exists because of old CRT monitors.
And for those in the audience who where kind of born after CRT monitors,
this is what they look like.
The way they work is, they have an electron beam
that scans across, highlighting the phosphorus.
This electron beam can’t just be… get back to other side of the screen
straight away, or get back to the top of the screen. And so these periods
where we are transmitting Control data was to allow the electron beam
to get back to the location where it needed to start
transmitting the next set of data.
That’s why it exists. Why do we care?
Because the encoding schemes for Control and Pixel data
are actually quite different.
This is the main difference. I’m going to come back to this slide
a bit later. But again, the important thing to see here is
that despite the encoding scheme being quite different
the output is 10 bits in size.
That first step – choosing whether it’s Pixel or Control data –
is described by this bit of the diagram. You might notice that’s
not the first thing in the diagram.
How do you convert Control data to Control symbols?
First we need to know what Control data is. There are two bits,
there is the HSync and the VSync signal.
They provide basically the horizontal and vertical pixel sizes.
They are kind of left over from VGA. We don’t actually need them
in HDMI or DVI to know where the edges are
because we can tell the difference between Control and Pixel data.
But they kind of still exist because of backwards compatibility.
This means that we have two bits of data that we need to convert to 10 bits of data.
So, a 2b/10b scheme.
How they do it is they just hand-picked four symbols that were going to be
these Control data symbols.
These are the four symbols. There’s some interesting properties with them.
They are chosen to be DC-balanced. They roughly have the same number
of 0s and 1s. So we don’t have to worry about the DC bias with these symbols very much.
They are also chosen to have seven or more transitions from 0 to 1
in them. This number of transitions
is used to understand the phase relationship
of the different channels. So if you remember this diagram,
we have a cable going between the transmitter and the transceiver.
These are, again, very high speed signals.
And even if the transmitter was transmitting everything at the same time,
the cable isn’t ideal and might delay some of the symbols.
The bits on one channel [might take] longer than others.
By having lots of these transmissions we can actually find
the phase relationship between each of the channels and then
recover the data. And so that’s why these Control symbols
have a large number of transitions in them.
More on that later when we get to the implementation. And I’m running out’ time.
This part of the diagram is the Control data encoding.
*sip from bottle*
What about Pixel data and the Pixel symbols?
Again, in DVI each channel of the Pixel is 8 bits.
And the encoding scheme is described by the rest of the diagram.
But again, it’s actually really, really simple.
This encoding scheme is called 8b/10b,
because it takes 8 bits converting it to 10 bits.
However, there is a huge danger here because IBM also invented
the 8b/10b scheme that is used in everything.
This is used in DisplayPort, it’s used in PCI Express, it’s used in SATA,
it’s used in pretty much everything on the planet.
This is not the encoding TDMS uses.
You can lose a lot of time trying to map this diagram
to the IBM coding scheme, and going these are not the same.
That is because they’re not the same. This is a totally different coding scheme.
Encoding Pixel data is a two-step process. I did say it was three-ish steps
to do this. The first step is we want to reduce
the transitions in the data.
How do we do this? – Sorry, why do we do this?
Because this, again, is a high speed channel.
We want to reduce the cross-talk between the lanes.
They are actually quite close to each other.
So by reducing the number of transitions we can reduce
the probability that the signal propagates
from one channel to the next. And how we do it?
We’re gonna choose one of two encoding schemes.
An XOR encoding scheme or an XNOR encoding scheme.
How do we do the XOR encoding scheme? It’s actually pretty simple.
We set the Encoded Bit same as the first Data Bit
and then the next Encoded Bit is the first Encoded Bit
XORed with the Data bit.
And then we just repeat until we’ve done the 8 bits.
So this is how we do the XOR encoding.
The XNOR encoding is the same process, except instead of using XOR
it uses XNOR.
How do we choose which one of these to use?
If the Input Data byte has fewer than four 1s
we use the XOR. If it has more than four 1s we use the XNOR.
And then there’s a tie-break (?) if you have even.
The important thing here is that this method is determined by the Data byte only.
There is no hidden state here or continuous change.
Every pixel has a one-to-one mapping to an encoding.
Then we append a bit on the end that indicates
whether we chose XOR, XNOR encoding of that data.
And so that converts our 8 bits Input Pixels
to 9 bits of encoded data, effectively our 8-bit encoded sequence
and then one bit to indicate whether we chose XOR, or XNOR encoding
for that Data bit. So that’s it there.
This encoding is actually very good at reducing transitions.
On average, we had roughly eight transitions previously,
now we have roughly three-ish, so it’s pretty cool.
I have no idea how they figured this out.
I’m assuming some very smart mathematicians where involved
because discovering this is beyond me.
And that describes the top part of this process.
*sounds of scratching nose and beard*
This is where, in the TMDS, the Transition Minimization comes from,
that step there in the encoding process.
But there is still one more step.
We need to keep the channel DC-balanced, as I explained earlier.
How can we do that? Because not all pixels are guaranteed to be
at zero DC bias like the Control symbols are.
We do it by keeping a running count of the DC bias we have,
and then, if we have a positive DC bias
and the symbol is also positively biased, we invert it.
Or, if we have a negative DC bias and the symbol has a negative DC bias,
we invert it. And the reason we do this is
because when we invert a symbol we convert all the 1s to 0s which means
a negative DC bias becomes a positive DC bias.
As I said, we chose – because we are already negative and the thing was negative –
we convert it to plus. It means we’re going to drive the running DC bias value
back towards zero. We might overshoot, but the next stage
we’ll keep trying to oscillate up and down, and on average over time
we keep a DC bias of zero.
And as I said. Then, to indicate whether or not we inverted
or kept… the… straight through we inverted,
we add another bit on the end. So that’s how get our 10 bit
encoding scheme. We have the 8 bits of encoded data,
then one bit indicating whether or not it used XOR/XNOR encoding,
and then one bit to indicate whether or not we inverted the symbol.
That describes this bottom part of the chart.
Now you can see partly why this chart is kind of confusing.
It’s no way in what I think of a what’s a logical diagram.
This might be how you implement it in hardware if you really understand
the protocol, but not a very good diagram for explaining what’s going on. And…
*sip from bottle*
As you see it’s actually pretty simple.
In summary this is the interesting information
about the two different encoding schemes.
Because we minimized the transitions in the Pixel data
we can actually tell Control and Pixel data apart
by looking at how many transitions are in the symbol.
If it has six or more transitions it must be a Control symbol.
If it has four or less it must be a Pixel symbol.
You now know how to encode TDMS data
and how to decode TDMS data
because if you want to decode you just do the process backwards.
Congratulations! How do you actually implement this?
You can just write the XOR logic
and a little counter that keeps track of the DC bias
and all that type of thing in FPGA.
I’m not going to describe that because I don’t have much time.
But if you followed the process that I have given you
it should be pretty easy.
But… and this is what we use currently.
You could actually use a lookup table. What we are doing is
converting 8 bits of data to 10 bits of data.
That is a lookup table process, pretty easy.
FPGAs are really good at doing ‘lookup table’-type processes,
and it also allows you then to extend this system
to those other protocols like the 4b/10b that is used
for the Auxiliary data.
So we are looking at that in the future. It uses a few more resources
but it’s a lot more powerful.
This is kind of what your encoder will look like, and your decoder.
It’s quite simple, it takes in your 10 bits of data
and outputs either your 8 bits of Pixel data
or your 2 bits of Control data and the data type.
This is kind of what if you went into our design and looked at it
at high level, in the schematic,
you’d probably see a block that looks like this.
The encoder is slightly more complicated because you also have the DC bias count
that you have to keep track of. But, again,
the data goes in and the data comes out.
That’s simple, right?
This kind of extends to Auxiliary data, or if you get an error,
like if you… There are 124 symbols
that you can have in 10 bits of data.
Not all of them are valid. So if you get one of these invalid symbols
you know you have an error.
However, things happen quite quickly
when you times them by ten. And so our Pixel Clock
for 640x480 is 25 MHz. When you times that by ten
you get 250 MBits per channel. When you’re doing 720p
you’re doing 750 MBits per channel.
And 1080p is at 1500 MBits per channel.
An FPGA is fast, but they’re not really that fast
at a range that I can afford to buy. I’m sure the military has ones
that go this fast, but I’m not as rich as them.
But they do include a nice hack to solve this.
They are called SerDes. They basically turn parallel data
into serial data.
This is what the boxes look like.
You give them your TDMS parallel data
and they convert it to high speed serial data for you.
They are a little bit fiddly to use and your best option is to go and find
a person who has already configured this for your FPGA
and follow what they do.
“Hamster” – Mike “Hamster” Field – has a really good documentation on
how to use these in a Spartan6. These are also unique to your FPGA,
so different FPGAs are going to have different control schemes.
But if you are using a Spartan6
then go and look up what Mike “Hamster” Field is
doing for configuring these.
Remember how I said, our system has a serial console.
Because we have that system we can actually delve quite deep
into what’s happening internally in the system.
*sip from bottle*
And print it out. This is debugging from one of our systems.
You can see…
The first thing is the phase relationship between each of the channels.
The next one is whether we’re getting valid data
on each of the channels and then we’ve got the error rate for that channel,
whether all channels synchronized, and then some resolution information.
You can see that this has got a 74 MHz Pixel Clock.
There are three columns because there is Red, Green and Blue channels.
This give us some very interesting debugging capabilities.
If you plug in a cable and you’re getting errors
on the Blue channel and nowhere else
it’s highly likely there’s something wrong with that cable.
This is a very powerful tool that allows us to figure out
what’s going wrong in a system.
It’s something you can’t really get with the commercial versions of this.
But what about errors? Everything I’m talking about now
is a little bit experimental, we haven’t actually implemented this.
But it’s some ideas about what we can do because we now
have complete control of our decoder.
As I said, there’s 124 possible choices for 10 bit symbols,
of which 460 are valid Pixel symbols,
4 are valid Control symbols and 560 symbols
should never ever be seen no matter what.
That’s like 56% of our space that should never be seen.
But it’s actually better than that! We know because of the running DC bias
that there are 256 valid Pixel symbols
at any one point. You can’t have the… if you’ve got a negative DC bias
you can’t have a Pixel symbol which continues to drive you negative.
Actually, 74% of our space at any time
is not allowed to exist.
This means that a huge number of the invalid symbols
are only near one other valid symbol.
And so we can actually correct them! We can go: “This symbol must have been
this other symbol, because it’s not a valid symbol,
it must be a bit error from this other symbol.”
So we can correct these errors. This is quite cool.
We can correct about 70% of
single bit flip errors in Pixel data.
But sadly there is some that we can’t.
But we can detect that we got a invalid Pixel data.
So the fact that there is an error is important.
In this case we’ve got two pixels that we received correctly
and we got a pixel that we know is a invalid value
and then two more pixels that we received correctly.
You can imagine this is a Blue channel,
so the first ones were not very blue.
Then there’s the decoded value for this is
very, very blue, like very light blue and then some other ones.
This looks really bad, right?
This was probably a whole blue block.
One pixel difference of that big, that size,
is probably not a valid value,
and so we can cover them up!
We can go… the two pixels on either side
and average them and fix that pixel.
This allow us to correct a whole bunch more of errors that are occurring.
And as we’re about to take this data
and run it through a JPEG encoder
this doesn’t actually affect the quality of the output
all that much and allows to fix things that would otherwise
be giant glaring glitches in the output.
That’s some interesting information about
how you do TDMS decoding and how we can fix some errors.
The thing is, we can do it even better than this
because it’s an open source project.
Maybe you have some idea about how we can improve
the SerDes performance. Maybe you have an idea about
how to do TDMS decoding on much lower power devices
than we use. It’s open source! You can look at the code
and you can improve it. And we would love you to do it!
The thing is that I have a lot of hardware but not much time.
If you have lots of time and not much hardware,
I think I can solve this problem.
These are links to the HDMI2USB project
and the TimVideos project; and all our code, our hardware,
everything is on GitHub under open source licenses.
And here is some bonus screen shots that I wasn’t able to fit in other locations.
You can see these small errors.
That one was kind of a big error.
This is what happens when your DDR memory is slightly broken.
Yeah… but – yeah!
And that is my talk!
*applause*
Herald: Excellent! Thank you very much, mithro.
As you’ve noticed, we have a couple of microphones standing around in the room.
If you have any questions for mithro please line up behind the microphones
and I will allow you to ask the questions. We have question from the Internet!?
Signal Angel: Yes, thank you! Do you know if normal monitors
do similar error recovery or hiding?
Tim: I know of no commercial implementation that does
any type of error correction. The solution for the commercial guys
is to effectively never get errors.
They can do that because
they don’t have to deal with the angry speakers on the ground
going wild as my slides look weird.
And, as well, they are probably working with better quality hardware
than we are using. We’re trying to make things as cheap as possible.
And so we are pushing the boundaries of a lot of the devices we are using.
So we are more likely to get errors than they are.
Herald: We have quite a lot of questions. Remember – questions, not comments!
Microphone number 1, please!
*rustling sound from audience* *coughing*
Tim: Yes!
*unrecognizable question from audience*
Sorry, I don’t quite understand what’s going on! *chuckles*
Herald: Do we have a translation?
Voice from audience: Audio Angel!
Tim: Audio problem?
*Herald speaks to person in front of stage in German*
Tim: I’ll be around afterwards, If you want to chat to me, ahm…
Herald: And we might do that… ah… write you on the computer afterwards.
Second question from microphone number 3, please!
Question: Hello? Ah, yes. Can you determine the quality of a HDMI cable,
e.g. by measuring bit error rate of each three pairs
and also some jitter on the clock, and that kind of…?
Tim: Yes we can!
The quality of a HDMI cable should be they’re zero bit errors.
So anything that has non-zero bit errors we chop up and throw away.
This gets interesting when you have very long cables.
We can actually see that the longer the cable is
the harder for them to keep zero bit errors.
So yes, we can kind of judge the quality of the cable.
But it’s also hard because
it depends on what the sender is doing.
If the sender is of a lower quality
and the cable is low quality you might get bit errors.
But if the sender is of a high quality
and the cable is of a lower quality
they might cancel each other out and still be fine.
We can’t just go: “This is a good cable”
because we don’t actually have any control over our…
how powerful our sender is on this device.
If we could kind of turn down the sender
and see where things start going wrong that would be pretty cool.
If anybody wants to look at building such a device
I’d love to help you do that.
Herald: We have another question from microphone number 5.
Question: Your hardware, the HDMI2USB hardware…
Tim: Yes! Question: Is it available for simply ordering
or has it to be solder soldered by hand, or…
Tim: You can not solder this board by hand unless you are much, much better
than I am. It uses Ball Grid Array parts because it’s an FPGA.
This is one here. You can buy them.
We’re working with a manufacturer in India who builds them for us.
We work with them, and it was pretty awesome.
We’re also working on new hardware. I’ve got a whole bunch
of FPGA hardware here that you can come and have a look at
and I’ll probably move it out into the hallway afterwards.
Again, if you’re interested in the hardware and you have a use case,
chat to me! Because I like to solve problems
of people who are not having hardware and my employer pays me too much.
So I get to use my discretion refunds
for helping out people doing open source stuff.
*applause*
Herald: We have at least four more questions. Microphone number 2, please!
Question: Do you think it would be possible to get an 1080p image
out of the open source hardware board you use?
Tim: Yes, I do, but it requires us to do some hard work
that we haven’t had time to do yet.
And for us 720p at 60 frames per second is good enough.
The USB connection
is limited in bandwidth because we don’t have an H.264 encoder,
we only have MJPEG. If somebody wants to write us a open source, say, WebM
rather than H.264 encoder
that might start become more interesting. We also have ethernet, Gigabit ethernet,
on this board. It should be pretty ease to stream the data out the ethernet.
I, again, need help.
The ethernet controller works. We can telnet into the board
and control it via Telnet. We just need somebody to
actually connect the data, the high speed data side up.
We use it for debugging and stuff.
Mike “Hamster” Field again, really big thank you to him,
he is an amazing designer,
he built 1080p60 that is a little bit out-of-spec
but actually works really well on hardware
that is almost identical to ours. He also did the DisplayPort,
like a 4k-DisplayPort which we can do on our board.
If you only need one or two of the 1080p things
DisplayPort connectors can be converted to HDMI quite easily
and you can do that on them.
Yes, I think that’s possible, but again:
open source … hobbyist … need developers …
Herald: We’ll take one question from the internet!
Signal Angel: Thank you. Have you considered JPEG2000?
Tim: No, I’ve not. And the main reason is that I want to be a webcam.
I want to pretend to be a webcam. The UVC standard, which is the USB webcam standard,
does not support JPEG2000.
There’s no reason we couldn’t support JPEG2000 when connected to Linux,
like we could fix the Linux driver to add JPEG2000 support.
Again, I don’t know if there’s any good open source FPGA implementations
of JPEG2000? So, that’s also a blocker.
But if you’re interested in helping out – come and talk to me.
As I said, I would very much love
to chat to you and solve the problems you’re having
with getting-going. As well, we have t-shirts.
I’m wearing a t-shirt that we have, and I will send everybody who contributes
a t-shirt. Whether that’s fixing our website, helping on documentation,
helping people on IRC getting setup, anything.
You don’t need to be an expert on FPGA stuff to help out.
And we also are working on a little project to run MicroPython
on FPGAs. So if you’re really into Python and you like MicroPython
I would love to help you help us do that.
It’s kind of working. We just need more powerful (?) support. So.
Herald: We have two more questions from microphone number 1.
Question: So, is there some sort of dedicated processor on that board,
or do you use like a Microblaze in the FPGA?
Tim: We use an open source soft core. One of three.
We can change which soft core we’re using with a command line flag.
We’re using either the LatticeMico32
which is produced by Lattice Semiconductor.
We can use the OpenRISC-1000
or we can use a RISC-V processor.
We generally default to LM32 because it’s the most performance
for least FPGA resource trade-off.
But if you like RISC-V or OpenRISC-1000 better
for some reason, say, you want to run Linux on our soft core,
then you can do that. With a one line command line change, yeah!
We’re looking at adding J-Core support in early next year.
J-Core is quite big, though, compared to LM32. So,
it probably won’t fit on some of the very small devices.
Question: So it’s a Lattice FPGA?
Tim: It’s a Spartan6 FPGA. And our new boards will probably be Artix-7
But we’re still in the process of making them exist yet.
Question: Thanks. Tim: I’ve also been working with
bunnie’s NeTV2, porting our firmware to that,
which has been really awesome. He’s doing some cool work there,
and he’s kind of inspired this whole development by showing that,
yes, you could do this, and you shouldn’t be scared of it.
Herald: Good, one more question from microphone number 1.
Question: Yes. Do you have any plans for incorporating HD-SDI into your platform?
Tim: Yes and no! We have plans and ideas
that we could do it
but HD-SDI
and all of the SDI protocols are much harder for the consumer,
generally to access, and we want to drive the costs of this down
to as low as it can go. And…
HDMI is a consumer electronic thing. You get it on everything.
You get it on your like five-buck Raspberry Pi.
HDMI is probably a really good solution for this.
We haven’t developed any SDI cores or anything like that,
so I can’t tell you like that we’re doing anything there
but if somebody’s interested, again, I like to remove roll (?) blocks and
we would love to have people work on that.
Herald: We have one more question from the internet and we have two minutes left.
Signal Angel: OK, thank you. The question is not related to HDMI but to FPGAs.
FPGAs are programmed in a high level language like VERILOG or…
after simulation you compile. So every vendor has created his own compiler
for its own hardware. Are you aware of a move to open source compilers
or to independent hardware? And do you see a benefit in open source FPGA compilers?
Tim: Yes! If anybody knows
about FPGAs you know they use proprietary compilers.
And these proprietary compilers are terrible.
I’m a software engineer. If I find a bug in gcc
I can fix the bug. I’ve got those skills, and I can move forward or at least
figure out why the hell the bug occurred.
That is not the case with FPGA compilers. The FPGA compiler we use
is non-deterministic. You can give it the same source code and it produces
different output. I’d love somebody to reverse-engineer why that occurs
because I’ve removed all the randomness from random sources from it
and it still manages to do it! I’m really impressed. So,
Clifford has done an open source FPGA tool chain
for the Lattice iCEstick things.
He said he’s gonna work on the Actrix7 FPGAs.
Please donate to him and help him. I would… like…
if that exists I owe people who… like a bazillion of beers, because
the sooner I can get off proprietary tool chains the happier I will be,
and it will make my hobby so much nicer. So, please help him!
Herald: And do give Tim a big round of applause!
*applause*
*postroll music*
*subtitles created by c3subtitles.de in the year 2017. Join, and help us!*