Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/3029 Thanks!
[Music]
so what makes a physicist create a video
adapter for a 1989 Game
Boy apparently Tetris I was told um in
this talk Sebastian will uh take will
show us how he was able to create a
cartridge module uh that actually acts
as a USB video and uh with that yeah
Sebastian take it away I want I want to
see this
[Applause]
[Music]
yeah yeah thank you very much for the
introduction and thank you everybody for
giving me this stage uh that's
incredible um so um yeah what let me
quickly uh explain what you seeing here
because I brought a few live
demonstrations um so what we're talking
about today uh I mean the main title is
reconstructing game footage from the
game boys memory bus that's quite a
mouthful um the short title is the GB
Interceptor and that's um a project I
made an open source project um that
we're talking about which does exactly
what the main title says um what you're
seeing here on camera is actually a
heavily modded uh Game Boy but uh this
would also work with the original Game
Boy from my childhood um and it's pretty
much the same like my childhood because
my parents always impose a 1 hour limit
of playing to me so again have this one
hour limit feels just like coming back
to my childhood um but the reason I'm
showing you everything on this thing is
because that has been moded with a
modern uh display um instead of this
original thing which would be quite hard
to see for everybody but everything I'm
showing works with an original Game Boy
it actually has an original main board
inside okay then the device we're
talking about let me also quickly show
you um often is depicted like this this
is just the same thing I have here but
only with a 3D printed uh case that's
perfect um and the idea is now that um
anybody who wants to record or stream
something from their original Game Boy
their unmodified Game Boy of course um
can have this adapter in between and
record from it um yeah without modifying
the original hardware and maybe the
Beloved Game Boy they know from their
childhood it's not in the frame so I
just plugged in Tetris okay and um yeah
so oh it's actually not working when I'm
showing it oh yeah I just unplugged it
right so
sorry let's reconnect
this yeah there we go no it's not
working great usually it works otherwise
I wouldn't be
here um okay so while this is trying to
work again uh let me explain um why I'm
doing this okay yeah now it's working
um okay so um if you're somewhat
familiar with uh how Retro Gaming works
today you would know that of course
we've got emulators we could play on to
stream stuff and record stuff uh there's
something called a GB operator uh which
is somewhat an emulator that can read
the cartridge uh which coincidentally
has the same letters chosen not related
to any trademark yeah um and uh there
are of course more than Game Boy clones
like the uh analog pocket some people
might have heard of that one and um of
course there are HDMI mods that you can
apply to an original Game Boy and I
added the GB Interceptor to this list
and the quick reason is that someone
contacted me about Tetra so yeah that
part was correct in intruction uh
because they wanted to do an Tetris
tournament and everybody brought their
own Game Boy on which they trained they
wanted to keep their muscle memory
working on the original device and you
cannot ask everyone to mod their game
boy just for streaming so he asked me
about it and I came up with this and to
be honest I haven't streamed anything
yet except for yeah this talk now um for
me of course it's just curiosity and
figuring out how to make this work and
it was a lot of fun I tell you so um let
me explain how this is even possible
before we look into the details an
important part for this if you're
thinking about oh what couldn't I do
this uh with the PlayStation as well or
something like okay that one has an HDMI
out but um important part for this is
that the the Game Boy does not uh cach
anything so it directly reads from the
ROM so the game cartridge is just
external memory um it also can read from
this ROM in the cardridge as fast as it
can read from its internal Ram which
means that there is no reason for any
Game Boy game to load the code into RAM
first before executing it uh you only
have to do this if you want to modify
the code on the fly so most of the time
the code is directly executed from the
cartridge y this is the problem they
mentioned
so I try to talk louder um so yeah the r
the the Game Boy would simply execute
most of the car most of the code
directly from the ROM and um the entire
bus the memory bus is simply a shared
bus between the cartridge and uh
internal memory uh not entirely so there
is an dma operation which would which
would split things and um there uh for
most internal parts we cannot see the
data that's transferred but we can see
the address that the Game Boy is uh
requesting data from a writing to which
is also quite
helpful and so this means that most of
the time we can simply look at the bus
and see exactly what the CPU is doing at
that moment or which instructions it's
receiving um so in this talk uh let me
quickly give you an overview of the
hardware of the Jeep Interceptor and
then look into three details um how we
recreate the memory in there emulate the
graphics unit of the Game Boy and then
finally something that's more modern
topic but makes this device so simple to
use for everybody uh mjpeg encoding on a
microcontroller so we start of course
with the cartridge yeah
[Music]
[Applause]
so
okay so actually I had this still uh on
my my disc drive from a YouTube video I
met about this so but there will be
stuff specific for today um so if we
look at the 32 pins of the card first of
all there's a power supply for the
cartridge or voltage Supply which is not
irrelevant to us the GB Interceptor is
running off USB power but this helps us
to determine whether the Game Boy is on
or off simply to give some message for
the user the more important part um is
are these four pins specifically the
clock pin this gives us just a ledge
every time uh something is written to
the memory or read from the memory so um
it's a 1 MHz bus so 1 million times per
second we get um an edge here which tell
tells us we're writing or reading and
the other two pins tell us whether it's
a read or a write chip select simply
helps to the cartridge to tell that uh
yeah it's whatever is happening right
now is not it's none of its business
okay so um simply to ignore something
and then uh we have 16 pins uh making up
the address it's a 16bit address yeah so
this can address up to 64 kiloby and um
finally there are eight pins for the
data so simply imagine 1 million times
per second there is an edge on the clock
signal and then for the moment this Edge
comes these 16 pits have pins have to be
set by The Game Boy which give it an
address and depending on whether it's
reading or writing the cartridge has to
set these eight pins or the game has to
set these eight
pins now to um actually connect our
microcontroller to it uh I designed this
PCB which is first of far as I mentioned
open source yeah so uh design and
Y
um it's maybe noteworthy that this is
actually open enough and I intentionally
chose it that uh everyone who wants to
have one can buy it on AliExpress by
someone who's absolutely not related to
me and I don't earn scent from it but it
becomes quite cheap to get one of these
this way okay um but more importantly uh
the PCB then has this Edge over here uh
which looks just like the pins from the
carbridge because that end goes into the
Game Boy and over here these uh these
these points where can sld something on
um is where I have to solder the
cartridge um slot that's something you
can easily uh buy it's an aftermarket uh
part and you can simply so it on um on
this image it should be oh it's the
unassembled PCB um and those two are
electrically connected directly without
anything in between so that's important
even if uh I um do not plug in the
Interceptor or even if I took a
screwdriver and and pulled off the the
microcontroller the gamer would still be
a able to control uh to play the game so
um there's nothing interfering with the
game it's just as original as it gets so
to be honest the name GB intercept is a
little bit misleading it's not
intercepting anything so the better name
would probably be GB eavesdropper but I
think you agree that this sounds less
cool so yeah I went with a GB
Interceptor so the only interesting part
about this entire PCB is that I added uh
a few bust transceivers that allow me to
also hook up the signal from the data
bus to an
rp240 um to those who are not familiar
with the term rp240 that's the
microcontroller from Raspberry Pi it's
not a Raspberry Pi for those who haven't
heard about Raspberry Pi Pico so a
Raspberry Pi usually is a single board
computer with an operating system and
everything that's not what's in here the
Rasberry Pi Foundation also released the
Rasberry Pi Pico which is an actual
microcontroller um and um yeah so uh and
the rp240 is the microcontroller on
board of the um Raspberry Pi Pico and to
those who are familiar with the Rasberry
Pi Pico almost the entire design of the
GB Interceptor is taken from the manuals
of the uh Rasberry pie foundation so I'm
not that much of an electrical engineer
yeah so um this is mostly Rasberry P
Pico just in clone from their minimal
Hardware example including the
oscillator and voltage regulator and
everything and the only thing I added
are these bus transceivers that allow a
direct connection to the memory bus and
the reason that I picked the
rp240 can be seen in the schematics
don't worry you don't have to read
everything right now yeah so um I
highlight what I'm interested in and
they are these two little things here
called Pio that stands for programmable
IO um they go in between the CPU cores
and the GPO pins um so I hooked up the
memory bus to the GPO pins the cores
could theoretically read those pins
directly but the problem with this is
since it's a 1 MHz bus uh the CP puu
would have to react to an interrupt 1
million times per second or it would
have to observe the uh clock and that
takes a lot takes away a lot of cycles
and uh is somewhat too slow instead
these programmable iOS can take care of
that because they are simply for State
machines per program IO so in total we
would have eight State machines that are
directly hooked up to the G GPO pins and
for GB Interceptor we only need one of
these State machines uh this actually
only needs four instructions uh so they
are like saying little computers is
maybe a bit much because they are dump
as a brick yeah um they only send very
few instructions most of them are around
reading GPO pins and pushing stuff to uh
a queue um first in first out queue but
that's exactly what we need so I that's
it takes only four instructions for the
state machine to observe the clock from
the cartridge bus um and then read out
all 32 pins every time there's a clock
event and push it onto to the queue from
where the CPU can pick up each bus event
whenever it's convenient okay so the CPU
does not have to be interrupted it can
uh handle each bus event at its own time
and after that pick up the next event
the only limit is that it may not fall
behind more than the uh size of this um
uh of this queue um which I think is
like four instructions so it's not that
much but it can even out a few uh tight
spots so that's pretty much uh the
entire Interceptor and so now the rest
is software and we have three tasks
tasks that I have to talk about now the
first core of the um of the rp240 now
has to interpret these bus events that
were picked up from the programmable IO
um and recreate the video RAM of the
Game Boy because I will show this uh on
pretty much well in three slides um then
there's this part of memory in the Game
Boy which entirely determines what's
shown on screen and we have to get a c
Cy of that we cannot see this directly
on the bus we can see the addresses that
the Game Boy um tries to read from or
write to in this vram but we cannot see
its content we have to recreate it by
following the the instructions to the
CPU that we can see on the bus then the
second core has to emulate and that's
really just pure emulation emulate this
Graphics unit on a Game Boy that's
called picture Processing Unit or short
PPU I will use that term several times
um to render the image based on that
state of vram and then the remaining
sources and we have to talk about what
is remaining after that have to handle
mjpeg encoding and uh act as a USB video
class because that's the nice thing
about this uh device in the end from the
point of view of most Retro Gaming
enthusiasts uh because this can simply
be plugged in into most systems without
the need for any drivers because it
really acts like a video camera yeah so
um so let me quickly demonstrate because
you can quite literally literally use it
as a a video camera so some of you might
have seen this thing that's the gamberg
camera it's one of the first digital
cameras out there and certainly the
first one I ever owned it's also one
from my childhood um it goes in the
other way around because the Jeep
Interceptor um shows the other direction
and if I turn this on oh it's pointing
at you but don't worry there privacy
built in the resolution is so low
U so yeah um you can pretty much uh plug
this into uh any Linux system Windows
system Mac OS and actually also Android
and uh simply go into shoot mode of the
gam camera and you it's down here sorry
and you can simply you're still seeing
me
right so um it's maybe a little bit too
late after the pandemic when everyone
was doing video conferences but if we
have another video conference um that's
the way to go so um yeah and as I said
it's works perfectly with everything I
tried under Linux Windows and Mac OS on
Android the only limit is that it's an
external webcam Android supports
external webcams but most of the apps
don't most apps only think of the
internal cameras so um it's a little bit
difficult to find a decent app that
supports this but this is not a driver
issue but simply that nobody usually
cares about external webcams on a mobile
phone yeah okay so um now on to the
three things that I mentioned that we
have to talk about um the recre vram
from the bus events uh the PPU emulation
and mjpeg encoding so yeah I have to
start with a little bit of um Theory uh
but I keep it as little as possible um
because the internal memory of the or
actually the memory bus of the uh game
addresses pretty much everything mapped
onto the 16bit addresses so the lower
half of these 16bit addresses goes
directly to the ROM so this would be
something external that's something that
goes through the GB Interceptor that's
something we can see also some games uh
Implement external RAM which then would
also be visible to uh our little device
here but then there are things like the
vam that's the part that we are after um
that's not visible because that's an
internal part we can still see the
addresses that are addressed yeah but uh
we do not see the data same goes for the
working Ram um and there's something
called the um object attribute memory um
that's a little piece of 160 byte that
determines the location of the Sprites
on screen so this is also important to
recreate the image that's shown on
screen um so here I'm aware of that and
I need this also but during this talk
when I refer to vram I usually mean both
the vram and this little 160 byte piece
at the end that's also required to draw
something on screen um then it might be
worth mentioning that uh the io is also
an internal part so if I press a button
that's not visible on the GB Interceptor
that's an internal part
um where the addresses are also mapped
onto a specific register where the
gamble can read out which button has
been pressed as well as uh Link cable
and yeah there will be a slide about
this problem um yeah and so what we need
to do is we need to observe uh what the
CPU is doing and then for this upper
half of the memory maybe this one is not
required but uh it's also uh in there as
well we keep a local copy of the memory
in um uh on the rp2 40 so while we see
instructions for the Game Boy if Game
Boy receives an instruction write this
value into this part of vam then we
simply mimic this instruction and write
the same value into our copy of vam
that's the basic
idea um in terms of instructions um so
we have a 16bit uh memory address but an
8bit CPU uh these are the op codes uh of
the Game Boy um uh CPU um and yeah don't
worry I will not drag you through a
bunch of assembly today yeah but uh just
to give you an idea how what this looks
like um this is the beginning of what
the GB Interceptor sees when you start
the Legend of Zelda links Awakening uh
Game Boy games always start at the
address 10 but they almost immediately
jump to a higher address because there's
something else in the way there and so
what we need to do is we have to observe
the first part bit of data look the up
the up code that's trivia one that's no
operation nothing's happening here um
look at the next one C3 so C three would
be UPS over here that's a jump and now
there's a tricky part I want to show you
um this instruction uh takes three bytes
um to to um yeah to be explained to the
CPU uh but the CPU takes four Cycles to
execute it which means that after the C3
we see two bites that are part of the
instruction that's the jump address
where the uh where the CPU should jump
to what should be executed next so this
is the op code this is the data and then
we see a fourth bite because the CPU
takes one more cycle to execute the jump
um but the memory bus does not stop
there yeah we still there's still
something happening um you can see from
the address that the um uh game is
actually masking it outside of the ROM
space so that very simple games do not
react to it but besides that we simply
have to ignore that because that's just
junk that's happening on the bus while
the CPU is still busy it's not always uh
junk for those who are a little bit more
into these kind of systems here you can
see uh the St pointer being addressed
yeah but I don't want to go into these
details uh but most of the time it can
simply be ignored so for every game and
everything we do we have to understand
um which parts are the op codes which
parts are data and what is junk that can
be ignored uh and luckily we do not have
to do any decisions on our own so if the
Game Boy decides to do that jump we do
not have to emulate the um um program
counter so which address comes next we
simply see it the Game Boy is dragging
us through the game and every branching
every decision is done by The Game Boy
so we cannot actually do a wrong
decision yeah we can only display
something wrong on the screen which we
will see later but uh any branching any
if yeah um happens on the game board and
we simply see the result um only problem
with this is that these conditional
jumps are a bit trickier uh because if
we look up this madeup example um for
the address 20 that's a conditional jump
if the zero flag uh is not zero is not
set uh so um yeah typically a comparison
of a value yeah um and you see here it
takes two bites because the second bite
uh so this the op code this is the
second bite uh this means that we jump
five bytes ahead so that's quite trivial
but the problem is um if you look at 20
there it takes three or two cycles and
this depends on whether the jump has
been taken or not um and to determine
this we still have to observe the
address because otherwise wouldn't even
need the address because we could simply
follow the op codes that are coming in
but we needed the address to see okay
here the address has been incremented by
once so the jump was not taken and this
is the next instruction increment of
register B doesn't matter um or in this
case the jump was taken because the
dress did not increment but the Game Boy
nicely uh put uh the the higher bit uh
up to get it out of ROM space um and so
this is garbage that we need to ignore
so just as an example of a few of the
um details that you have to look into
when you try to execute the code that's
seen on the bus so I hope at this point
the idea is clear what we're doing um
intentionally I wanted to show a bunch
of edge cases but I realized that I'm
showing a bunch of assembly to you and
uh that I only have 45 minutes or 50
minutes something about that um so um
let me just show you a few of the tricky
bits uh and then mention the others uh
one problem can be that uh the um the
data as I mentioned before from the io
registers is not visible to the GB
Interceptor um and one example for this
is the diff register the divider
register it's pretty much a fast counter
it's simply counting up all the time and
that's used by many games um for random
number generation so imagine this
counter is counting up all the time and
at some point in the game the player
presses start and then that random
number or sequence of numbers that comes
from there is used to generate a random
level and the problem is that's not
visible 90% of the time s that's no
problem because that number is then used
internally by the game uh for a jump um
or conditional jump why is it not
working again it has stage fight has
it that's a little bit embarrassing if I
want to tell you how easy it is to use
this device so
um come on that looks better so yeah um
I can show you what I mean with the
random numbers uh I think most of you
are probably familiar with Tetris yeah
if you not familiar familiar with it you
should really try it it's great
um and um yeah so let me just start the
game so this is a typical example of
random data being generated that's
called the um the uh garbage stack so in
this play game mode you have to remove
these lines that have been generated
randomly and if you look closely you see
okay uh the layout matches up that's
because uh the random numbers are used
uh for a branch at some point where the
um where you
[Applause]
so the random numbers are used uh to
trigger a branch to decide whether
there's a gap I think it's because there
are multiple numbers mapped to gaps so
there's jump and the gep Interceptor
just executes the code that's
responsible for Gap but the design of
the garbage blocks is actually just an
offset calculated from the random number
and we do not know this offset because
that's just an arithmetic operation and
and if you look for example at the
bottom left um block here this one has a
different design than that one and
that's true for pretty much everyone
here so actually Tetris for which I
designed this has examples of where it
doesn't work that well uh because it's
uh cannot see the random numbers
generated or the the clock signal that's
used for Gen for random numbers that's
uh created internally um and
unfortunately um the tetris also has the
worst case example as well um because
the um link cable so serial transfer
register is also part of the io
registers so if you go into a two player
match uh of Tetris and um yeah use the
Link cable with the second game boy if
you've ever done that uh you might know
that if you see Mario then you were the
one who started the game so this is the
controlling Game Boy for the match and
if you see Luigi who's unfortunately
over second place yeah then you are on
the second game boy um and what's
happening if you start um if your Game
Boy is the controlling one then your
Game Boy generates a the garbage stack
and just like in the example before it's
you cannot see it here but you've seen
it in the previous example you have got
the same layout on the visible on the
Interceptor but the designs differ so
that's not that bad that's okay we can
deal with that but if you're the second
player then what happens is the first
game generates the garbage stack because
it should be the same for both game boys
and then uh those game boys become quite
lazy because the uh first game boy
simply transfers the entire data that
goes into vram and into work Ram through
the Link cable and the second gam writes
it just into its memory without doing
any branching or any checking on it
which means that to the Jeep Interceptor
nothing's visible there and yeah the
zeros you see is simply just the default
value of the memory at this point so
that's an example of where it really
doesn't work but trust me at the end I
will show you some statistics of how
many games are working that's an
exception yeah um yeah and as I
mentioned uh the same is true for the um
Gamepad input so um if you press a
button the Interceptor has no idea that
you press that button but the trick is
that that uh in again most cases um the
games simply read out the joyad register
um do a conditional jump if button a has
been pressed then execute the code that
makes Mario jump and so we have no idea
if that button was pressed but we simply
follow the code that makes Mario jump
because that's being executed so in most
case it doesn't matter there are some
few examples um where again um if bit
for the direction uh is used to flip um
to flip Sprites on screen to mirror the
player when he's moving I'm not sure why
they do this I don't think that it's
much faster this way because they're
doing conditional jump anyway to
accelerate the the player but luckily
that can then be fixed with game
specific fixes which I will mention or
show later again I just mentioned them I
will show them later um other tricky
bits which I just mentioned uh interrupt
detection Game Boy also has interrupts
that can occur at certain times and uh
that can be tricky um code execution
from Ram still occurs sometimes for
various reasons that's not as tricky
because we already created a copy of
that RAM and the code in Ram has to be
has to have come from somewhere and
that's something we have seen um really
trick dma transfers the um Game Boy has
one specific dma mechanism to copy data
to O memory and if there that happens
then uh the memory bus actually splits
and we cannot see the addresses anymore
so there's a dma unit that can for
example transfer data from the cartridge
and in that moment we see this transfer
not what the CPU is doing but um most of
the time the games luckily do not do
anything anything significant so most of
the time it's not much of a problem
that's actually a part where I found a
bug in a Game Boy game which I wanted to
show you it's in the abstract sorry I
figured I don't have the time for that
you can ask me later about it um yeah
and of course many little edge cases but
besides those that's the point where we
have a copy of the video RAM okay um so
the next thing we have to do is Graphics
emulation um and for this I first have
to give you again a short introduction
on how the graphics of the gamber works
if you want to learn more about the
graphics uh there's an amazing talk um
about uh how the Game Boy works by Mich
style from the uh 33rd uh C
communication Congress I will give a
link at the end yeah so uh that's my
suggestion so this is only basic idea of
what's happening there because the Game
Boy does not use a frame buffer like
modern device where would do where you
can simply draw everything the entire
image and then hand it over to the
graphics unit to be displayed but
instead it has to set up everything from
these tiles so this is most of the part
that's in the video that's what we are
after um and as you can see from the
footage from Zelda here recorded with an
emulator so we can see the tile data uh
the entire thing on screen is Simply
Stamped from these tiles from something
called a tile map which is simply a map
of these addresses um something that
Zelda doesn't seem to be using at this
point is also that This Ti map is larger
than this screen and there's a scroll
register so you can also scroll this
background map in in certain ranges and
on top of that there are some special
features like a window which is
responsible for drawing this uh this
status screen down here and um of course
Sprites which are called objects on the
Game Boy uh which are the players and
the enemies on top okay but it's all
created from these tiles and one other
thing that you need to know which is
actually the tricky part is that the
Game Boy renders uh everything on screen
like an old CT so line by line it draws
from left to right um and you now see
this gray area is what I have as a
screen and then eventually it will draw
everything else and the green areas that
I'm showing here is the part where um um
yeah actually internally the location of
the virtual C CRT beam because um The
Game Boy even the original display
didn't needs to be drawn line by line I
think it's probably because everything
at the time was uh written that way um
is where it's outside the screen
virtually yeah and this is important to
the games because while the PPU is
drawing something vrm is not accessible
so in this animation I had this little
block here so it's blocked while it's
drawing the image and then it's
accessible after that also for those who
are familiar with this I know this is a
simplification there are some more
details in timing and how long this uh
takes but the important part for the
games is that they have to wait for the
green Aras to draw something or to
change anything about the graphics or
most things of the graphics that's
called horizontal blank and that's
called vertical blank vertical blank
means there are 10 entire lines where
the CPU is not interrupted um by being
something being drawn on screen so
that's what what the most easy thing for
most games to change the entire screen
so you could change stuff during V blank
and then draw the entire screen change
stuff during V blank but of course
that's boring and that's quite limited
uh many games will change stuff during H
blank and that's very important to us
because um we need to know um the state
of the vrm for each line that's being
drawn because if something vrm is
changed to the next line it doesn't help
if we emulating rendering this line up
here yeah so we need to be quite exact
about the timing and I would say there
are three levels um of uh what can
happen there the first one is easy the
game changes stuff in vam uh in during V
blank um so um pretty much what you
would do nowadays you would change
everything while nothing is shown on
screen and then the next frame is drawn
and uh the timing in which the game does
this is about um 1 millisecond that's
easy I mean you've just seen that tetes
works so that's fine that's not much of
a problem the level two is when the game
changes something during Age blank um
and that could be changing some tiles
but also the typical uh example from
demo scene when you've got a logo which
wobbles because it's made up of the
background tiles and uh the scroll
register which I just mentioned um yeah
uh is changed so that things do not
align like this no it's the game boy
who's not
happy oh yeah okay yeah I you shouldn't
do this but come on we all want
[Applause]
to ah better okay um so yeah the example
those who are familiar with Game Boy
games uh know that this is an
game you see people reacting to this
this game is famous for its uh demo
scene like effects so this the effect
that I just meant the game is changing
the scroll register um during each blank
every time so let's do this one more
time um so this requires a Precision of
40 to 68 micros seconds really depends
on what's on screen that's what I meant
it's simplification what I had before a
come on
what's it's
really H okay no now this is
gone it's not nice if this happens on
stage I mean I've tried it so many times
now everything is gone that's
great and it's garbled on the Game Boy
as well just see that I'm bending
everything here so maybe I should take a
little bit more care while handling this
stuff okay let's try again although I'm
not sure why the camera is G gone as
well that's a bad sign but then I just
have to explain what you should see
sorry about that oh come on I tried it
so many times if I is that a good
idea did the camera go to sleep no
okay
hm okay no idea let's simply explain
what you're seeing ah I think this might
have gone
out now the
camera oh that's great now everything is
off okay well uh for those who are
wondering how this presentation
Works H now it
reloaded but at that point it should
trigger to reconnect to the camera okay
well too bad because the next example
would be the really exciting
one but um
something
okay okay uh because the next level
would be that exact game um which is
called prehistoric man and uh that game
um um so maybe I can hold it up to the
camera but then you don't see what the
Interceptor makes out of it I'm not sure
if those at least who watch the stream
can see it what that game does is um it
um does not only change the um change
everything during H blank it actually
also makes changes while things lines
are drawn on screen so actually during
the time where the video memory cannot
be changed um yeah maybe someone some of
those uh who stream this can see it so
there's text on top of it uh which
changes um while it's being drawn and uh
oh now you okay they just needed a
pause
okay okay
so let's do this
again now I just have to figure out how
to get back the time right so but so
let's show it again because uh then
hopefully it will work now uh but I hope
you believe me that this was more a
problem of the presentation technique of
being able to stream the camera than
anything so you see they're changing
stuff while the line is being drawn um
and what they're doing is they change
the color palette while it's drawn so
they cannot access vram but they're
changing the color palette on how the
tiles are mapped onto the four Shades of
Gray of the or four shades of green of
the Game Boy um and this means they're
changing the color palet every second
CPU cycle which requires a Precision of
1.9 micros seconds and as you could just
see that's something I cannot do on the
GB Interceptor I think the problem is
mostly um that uh the Precision of the
uh of the graphics emulation is not uh
there because I have to do some
shortcuts to be able to do this on a
microcontroller um and it's not that
much of a problem of figuring out how to
synchronize the thing yeah um yeah but
uh besides that usually timing is not
that much of the problem uh because we
have different mechanisms um oh yeah
if you want to learn more about pris to
man I recommend this block entry uh
sorry I have to go on check the
recording later if you want to see that
URL um because that game is really
interesting in terms of old school
Graphics so uh we have three means to
determine where the game is and it's
simply looking uh at the mechanisms that
the game uses to determine when it can
ride to video RAM um one mechanism is
simply interrupts the game has two
interrupts mechanisms that are relevant
for this either interrupt on entering V
blank or this uh interrupt here this uh
status register allows to set a specific
line when interrupt should be fired
unfortunately this is a little bit
slower than entering a tight loop on the
CPU and observing the stat register
directly um so if games have to be
really precise about this they will do
that that's trickier the gep Interceptor
can recognize many Loops because uh that
register has been read and then there's
a conditional jump right after that but
um yeah this is more or less free
programming that the uh developers can
do and there edge cases where this does
not work automatically and for the other
examples we need game specific fixes um
which means um I do I hash everything
that's written to video RAM and the
Interceptor comes with a large library
of hashes or Footprints um which are
created by running using an open source
emulator P boy was it um implementing my
hashing function and then emulating
1,600 games on my PC to set up that
database recognized games just so that
for the few games where this doesn't
work I can set up a specific address
where I know the synchronization once it
reaches that
address okay so yeah that's pretty much
uh all of the old school stuff because
now we have to handle uh you know we get
to have to get this into the modern
world modern yeah uh by doing um yeah by
by creating a USB video class device so
you already see this will be mjp
encoding um but when I created this I
did not even think that that would be
possible I thought about uh extracting
the data and reconstructing the game on
some host host software which would have
been a nightmare because you would have
to address all the operating systems
explain Windows users how to install a
python distribution and stuff like that
yeah so
instead um I eventually noticed okay uh
tiny USB the library that most rp240
projects use supports a video class
device um only problem is since I
designed this beforehand um I'm stuck
with the onboard controller which is
only USB 1.1 so what they call full full
speed uh USB B which is about 1 mb per
second in the isoc Chronos mode that I
used at the beginning for those who are
really into USB um but if you do the
calculations the game boy runs at 60 FPS
uh with a gigantic resolution of 160 by
144 pixels 2 bit per pixel that's four
shades of green yeah um and we could
easily fit this into the stream problem
is if you look into the specification
for the video class um we see a bunch of
compression algorithms which at that
time I thought were entirely out of the
question because yeah the the rp240
already has all its hands full um so I
looked into uncompressed video and there
are four color formats allowed and the
problem is those are actually color
formats so not two bits per pixel but
the best we can do would be nv12 with
12bit per pixel the problem here is that
we've got an 8bit gray scale image which
is already much more than we need and on
top of there there's a two times super
sampled color um or Luma Channel sorry
chroma CH channel on top of this uh
which is additional 4 bit per pixels and
if you do the maass the first version of
the GB Interceptor could only do 29
frames per seconds now the Europeans in
the audience would say okay that's fine
25 frames per per second were always
fine but the problem is that the Game
Boy um does these 60 frames per seconds
internally and that's a problem because
it uses this for some of the effects so
if we look at a game here that's um let
me just fill this up shortly that's
called called Bubble Bobble not sure if
anyone heard of the okay you know this I
have to admit I didn't know it beforeand
yeah so uh this I use this as an example
because uh it uses a lot of transparency
effects which require this refresh rate
so we have to look at the story it's
probably awesome um but you find the
same effect for example also in The
Legend of Zelda and in many other games
because the Game Boy if it needs to draw
transparency um it of course cannot do
so with only four Shades of Gray but you
see these bubbles here are somewhat
transparent and if we look at the proper
recording of this and slow it down by 10
Time by factor of 10 it does so simply
by turning them on and off in this case
it's flipping both sides of the of the
bubbles but could also be just entire
image flashing so uh on the actual
screen it looks like it's transparent
because it's faster than our eyes
actually on the modern moded screen you
might be able to spot it if it wasn't
for the camera uh but on the old screen
this uh look just like transparency
problem is if you do this with the 29
frames per second Jeep Interceptor it
looks like that yeah so this now real
time problem is not only that so it
would be okay if it was 30 frames per
second because the first Interceptor
already was able to do what I call frame
blending it could mix two frames and
therefore create the transparency but
with just 29 frames per second once per
second you slip to the other half of the
of the image and it just looks weird or
flashes once per second and uh it does
it's not nice so yeah the rro
enthusiasts wanted 60 frames per second
so what could we do I looked at what
mjpeg would uh require so let me give
you a very quick introduction of what
what mjpeg is mjpeg is just a sequence
of jpeg images that's really simple
there's not much more to it um and you
need five steps to encode in jpeg image
first step is convert RGB to a grayscale
and color channels uh Luma and chroma
that's trivial because we already have a
grayscale image and don't care about
color
and also nicely jpeg supports purely
grayscale images so that's good so we
can simply throw away the rest then you
have to split the image into 8 byed
blocks which is not too difficult but
can be annoying if you have to address
the pixels in different sequence on a
microcontroller uh on a 32-bit
architecture which encodes four of the
pixels in one bite especially um and
then you need to calculate the discrete
cosine transform um to my fellow
physicists and mathematicians that's
similar to for transform quite similar
actually and you can simply calculate
the coefficients through recreate the
image to everyone else who hasn't looked
into this yet the idea is that you make
up each 8 by8 tile of these 64 tiles let
me demonstrate so I can reset this to
zero and if you for example mix this
tile with this tile you get some kind of
a diagonal and if you turn this one on
as well then the black part is actually
negative numbers but you can show this
with colors this cancels out the other
part and you get something like a
quarter Circle and this way if you take
any from this image you see different
superp positions of these tiles to
recreate the image okay um it's also
nice for these ones when only the
horizontal ones show up for example yeah
um that's actually not um so yeah so the
problem with this is you need to
calculate these coefficients and yeah
that's not that fast you can use a
lookup table for these tiles that's okay
but still the calculation is much more
than the capacity that I would have left
on the rp240 and by by the way uh just
um yeah uh for your information um this
is not yet where compression kicks in
this is not lossy so so this could be
perfectly reconstructed the compression
is the next step the quantization where
you decide that the eye is not as good
as sporting um imprecisions on the
higher frequencies down here and um then
you quantize that and um accept errors
here and if you do that then you get the
typical artifacts that we all love and
know from jpeg yeah so that's the actual
part of the calculation if you have the
discrete coine transform that's probably
not that hard as well because um yeah
it's just a quantization but the
discrete coign transform is something we
already gave up in the end you also have
to encode this um there are two options
that are coming for this type of project
would usually be halfman coding which I
will not explain here so the idea is
that you have a Vari number of bits that
you can use shorter strings of bits for
more common numbers and longer parts of
bits for the more rare numbers which is
also really an aning because you have to
do a lot of bit shifting and um uh and
bitwise manipulation to fit this into
your 32bit blocks if you really do this
on a so a PC doesn't care about this but
a microcontroller can be stressed out by
this so I gave up on this and I thought
okay that's impossible I don't have any
resources left this thing is at its
limit when someone suggested something
on GitHub so katsuchi I hope I pronounce
it correctly also known as PL on Twitter
someone's laughing okay um suggested
that we could simply blow up the image
by a factor of eight and that way we
could get a compression ratio of 5 bit
per pixels okay how would that work
let's have a look if we increase it by a
factor of eight we get an image that's
1,280 by
1,152 pixels that's almost full HD so
not that promising right but let's look
at what we have to do then so converting
RGB to uh gray SC that's trivial that
has been trivial before that's fine
Split Image into 8 by8 blocks that's
especially trivial now because each 8
by8 blocks is now one of the pixels that
we just
rendered and uh then calculating the uh
discrete coine transform um is now also
trivial because if you look back at our
little toy here um a homogeneous block
is just that top left um tile that's
called the DC component the other ones
are called the AC component and here we
just need to adjust the brightness and
that's our homogeneous tile the other
ones we don't care about them we do not
have to do that calculation at all we
only have to encode this DC tile now so
that's trivial now um quantization is
especially trivial now because we only
have four Shades of Gray actually if we
do if we allow for frame blending that's
seven Shades of Gray because of the
mixes that can occur that's still the
limit of quantization that jpeg allows
for and the encoding again that's
something I don't have time to explain
now today uh but uh he was right his
example had very different number of
bits I changed it a little bit so that
we have exactly five bits per block and
that now means um that we can encode
everything with only these two lines
that have to be executed on the CPU
sorry know it's annoying to read code
like that but
the
um if you are really that quick with
that line then I we probably should
switch positions but um okay the the
thing in jpeg is that uh the pixel are
Delta encoded so this means the
difference from the previous pixels
encoded which is the only reason that
this is so complicated each 32bit
integer represents four pixels so I have
to shift them by one block and take the
difference and take one of the pixels to
the next iteration so these two lines
encode four pixels at the same time and
then they are written to the
programmable IO and that's where all the
remaining Hardware of the rp240 is used
because the rest is done by the
programmable iOS and chain dma even if I
wasn't running out of time I did not
intend to go through all the details
here but the quick idea is the CPU uses
these two lines to write four pixels
each in total 32 pixels split to two
State Machines of the programmable IO
and from there the result does not the
CPU doesn't even have to care about the
result the result is taken up by chain
dma that can be triggered by the program
IO and the end of the dma operation can
trigger the next dma that is what makes
a chain and they can split it onto two
more State machines each and in the end
everything is written to block and RAM
which can directly be served to the user
so there are one or two lines that are
required to trigger the final chain team
a um simply because of the how they
align and cannot be triggered
automatically but in the end this means
that really the CPU only has to once in
a while shove a few pixels up into this
pipe and everything else simply is
written to Ram by the hardware and
that's um yeah how we can achieve this
only problem is 60 FPS fits in there but
then we look again into the
specifications and it tells us you have
to have a color Channel
so we have to do two times subsampling
and um yeah so unfortunately there are
now two modes for the GB Interceptor the
mode that works everywhere which offers
30 frames per second and a mostly unused
um um color Channel and um you can have
you there's a button you can switch it
to Green if you like that yeah but you
could also do this in post and a mode
for 60 FPS for the kind of host software
that accepts jpegs that are only
grayscale which is easy for most host
software for example I just created a
very short pull request for OBS so that
the current version of OBS also accepts
the 60 FPS version in all three uh
operating systems okay and so this way I
can show you
now a come
on again the Amazing Story come on H um
but yeah you can imagine what it will
look like so this is now the 30 FPS
version with frame blending of course
yeah but if you can see it it looks just
like on the screen and this is now mjpeg
encoded which also helps with
compatibility and that's how I get to
the final slide summary and uh some
hints for further information so summary
is mostly compatibility chart um so far
as people have reported because I don't
own everything I've got a few game boys
here that you can try later but um it
works on pretty much every Nintendo um
device that can run original Game Boy
games but only grayscale games just to
be sure even if it says Game Boy C
Game Boy Color or Game Boy um Advanced
um that's only if you plug in a GRC game
in there it doesn't work on the analog
pocket simply because the uh how
interrupt detect the uh interrupts
there's one little difference in the
timing I think it's easy to fix I
promise it a year ago to fix it but I'm
a little bit worried to break something
else if I do that uh and some exotic
aftermarket thing called The Game Boy
Color um I don't own that I know don't
know why this doesn't work um in terms
of games um we've got a total and time's
up but I'm always done uh we've got a
total of almost 200 games or more than
200 games including hybrid games and
special cartridges uh most of them work
from those report working there are some
with minor glitches I count Tetris among
them so if you got seen this example and
only four classical games and one
special cartridge the special C
cartridge again I don't own that I'm not
sure what's going on there but the
classical games they are just glitching
that much I would say it's unusable and
for host software I said Linux windows
and ma OS everything I tried worked on
Android there's a problem that the few
apps that support external cameras are
not really that those high quality apps
and two of them worked one didn't but as
I said it should not be a driver issue
because it's an mjpeg external webcam
yeah and with that so um as I mentioned
if you want to learn more about how the
game Bo works I highly recommend the
ultimate game Bo talk by Michelle style
you find it in the media uh library of
the CCC of the 33rd um car communication
Congress for the technical details have
a look at gbde iio that's a public
domain repository of all the people on
Whose shoulders I'm standing who could
learned everything about uh the Game Boy
so there's everything is in there that
you want especially the pendock in there
and more stuff from me uh I've got a
block at there ought to be specifically
Jer can be found at there ought to be a
Game Boy capture cartridge um just note
this one is before MJ so it mentions the
29 frames per second limit you can find
me on masteron and other social media
channels but this and YouTube is the
main most important ones GitHub
repository over there and since I'm
running out of time just the last
announcement um you can later at 10 p.m.
meet me at the house of tea opposite of
uh um uh zal a um and if you want to
chat about anything details of this uh
if you just want to touch the wooden
Game Boy that's okay as well um so I'll
be there and now so thanks sorry for
being a little bit long
and
[Music]
what thank what an amazing what an
amazing labor of love I'm sure there are
going to be questions uh we have four
microphones in the room and we have a
signal Angel who already has a question
for US Signal Angel go
ahead yes thank you so much first of all
the internet was very impressed with
your talk there's a ton of great
feedback here um can you clarify if the
60 FPS supports now means that you have
a lossless captured result on the host
machine yes uh except of course for the
mistakes that Interceptor could do okay
so so just this asteris thing that you
saw Tetris that the wrong tiles are
shown uh that of course is not fixed by
this but this mjpeg encoding I didn't
mention it this is lossless because the
DC component uh the quantization is less
than the uh individual pixels so this is
lossless you get every single frame
there might be an exception if there is
the I'm not sure there are some times
where the scen screen can be turned off
by the game there might be something
that's visible on the GP intercept
that's not visible on the game board but
it's besides those edge cases in the
regular case you should get every single
frame and lossless the mjp part does not
introduce any uh compression artifacts
or
anything microphone one my wife hi uh
have you tried kind of backwards
guessing the values that they cannot
observe for example if you read the
value that you can't observe and then
take a branch based on that value you
can see the branch was taken so you can
make assumptions about that value to fix
the graphic glitches yes uh many of the
game specific glitches work that way but
they're quite simplified because of the
limited resources that I have available
so what I cannot do is so in theory of
course you could track of everything
track keep track of many possible
branches and when branch is taken uh
decide which branch was actually taken
to uh reverted back but um I only have
the simple version because resource
limited Al I'm at the limit of the ram
of uh the rp240 with what I'm doing um
is that I simply Implement specific uh
addresses and when code reaches that
specific address I know that a specific
value was there so for example when I
mentioned that uh the gamad inut input
is unknown and uh Sprite has to be
flipped I simply have a short uh yeah
entry in the in this um game database
which tells uh the Interceptor if the
game reaches this specific address then
this bit has to be set and that makes
the uh figure on screen correct but
there's no automatic way because there
are no resources left on this device
maybe I at some point I have to do a new
version with a faster processor but um
yeah all right thank you so microphone
three thank you for your great
presentation uh I have a question about
uh not the content of your presentation
but about your presentation itself with
with magic you have created this great
PowerPoint uh this great presentation or
the great slides uh well the tool I used
to create it is called a text editor but
um the the way I presented is that's
just a web browser so this is an HTML
page with some ja JavaScript and
um also works on pretty much every
operating
system thank
[Applause]
you microphone 2
please have you tried um cheat modules
like Game Genie or game Shar um yes um
it looks a little bit ridiculous if you
do that uh so I've got one here you want
to see it
right I don't know any cheat any codes
or anything so I cannot run it right now
but I tried it but um it starts with
questioning how which way it has to go
in oh yeah of course you have to put in
the Jeep Interceptor close to the Game
Boy because uh it needs to see what goes
through this thing and not what the if
the game genie changes after the GB set
seen it doesn't work which then means
this goes on the wrong way and then in
the end uh this has to go this way I
think because it was always wrong in the
game genie and yeah so I do not
recommend this yeah but it works I tried
it
yeah oh I will remove it now because I'm
mechanically worried about my game boy
so microphone one hi very amazing talk
thank you very much but Sebastian what
about
audio yeah oh I should have expected
that um after ADM in the first version I
ruled out audio simply because um I was
happy that I could even achieve that and
I thought that anyone who wants to
stream or record this can simply plug in
something to the audio jack so that's a
simple solution of course course um but
I've been thinking about it and I think
it's not that tricky if you don't want
to have an perfect representation
because this chain dma which I didn't
know when I started the project um can
be quite useful to uh jump in lookup
tables uh to for example reproduce sign
functions um stuff like that so maybe
few things can be done but I'm at the
moment with the mjp encoding I'm also
using up many of these addition
resources so I think there just is not
more room in there I have it on my to-do
list to at least look into it um but I
don't think that there's much I can
still squeeze out of it so until then I
think there's one uh Game Boy advanc
which does not support this but for
everyone else sorry it's the old school
audio check thank you um so we're
starting to run out of time for Q&A
microphone one again very briefly hi
awesome talk thanks um so what's the
video latency minus what the USB host is
doing um sorry I don't have the number
in my head um I once made a Q&A video
where I tried it with the old version
with nv12 um and it was better than
expected but something I wouldn't play
on
um I have to admit I don't have the
number now I should check it with the
mjp version because it became quite low
but probably nothing for True retro and
true enthusiasts who prefer fpga systems
over emulator so um if that's the
question is aiming at that I can play
while watching the screen but I think
those who really Master action fast
action games uh would not be happy with
it okay okay thanks microphone four
thank you um has this kind of um video
streaming out of the rp240 also been
tried for other video
formats
um are you aware of any such projects
not that I know of also I'm not sure so
I got this suggestion from this guy on
GitHub so that's why I referred to him I
could not really find if something like
this has been done before but it's you
also have to remember that this is of
course very specific to the Game Boy uh
to this specific image that we have on
the Game Boy you can throw this
technique on any kind of image that you
want to encode so this is quite limited
to gray scale this High um um
quantization and of course if you want
to encode a larger image like that
remember that the host has to deal with
the image that has been scaled by a
factor of eight that's fine for the
small image but if you try to encode um
a full HD image this way maybe the
microcontroller can keep up with it
because it's simple task for it but the
host software will blow it up by a
factor of eight and probably run out of
memory that was reserved for this before
it can do anything with it so um I'm not
sure if this can be transferred to too
many projects but I would love to see it
um and I'm not sure if somebody has done
something similar before well well even
cooler if you're the first
one so with that I think we're out of
time and we're out of questions which is
a amazing coincidence let me well help
me by thanking Sebastian for this
amazing talk and effort thank
[Applause]
you
[Music]