Dear viewer, these subtitles were generated by a machine via the service YouTube and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/3196 Thanks!
[Music]
thank you very much for the um kind
introduction um yeah um I like Computing
and uh today I want to talk about one
fam one family of computing devices and
I've grown de to in basically the last
year and just tell you what experience a
and others had with them since they are
kind of obscure so
I'm uh before I'm explaining what a
vector computer is how about we just
narrow down what a computer is for the
sake of this discussion or this um
presentation uh computer has functional
units functional units get uh words from
registers and on these words they
perform some often mathematical
operation and write back a new word to
the registers registers are in constant
exchange with the memory where where
where things get loaded into registers
or stored back into RAM and all this is
controlled by an instruction decoder and
all kind of other stuff which is not
that important today so that is what I
that's what I mean with a computer in
this presentation so this raises the
question what is an instruction an
instruction asks who who gets processed
using what operation does um the
information get processed and where to
does it get stored where does it end
up so um what are words for computers
and for like visualization sake I
decided a cube is 46 bit for this
because well we have um if we um if we
imagine that every single cube in there
this 4x4 Cube could be on or off we end
up like with 46
different bit states which could be um
things which is the same as a 46 bit
word and like a 32 bit word would be
like the half of that like one dimension
half um and a word itself is not enough
to draw meaning from we also need a data
type to describe how to interpret the
word because the same bit pattern can
mean different things in different data
data
types so uh what is a vector computer um
this kind of is the idea of a vector
computer a pallet um collecting stuff in
the central place and being able to
transport it so rewriting the same graph
with that um metapher um a vector
computer um is not about carrying every
single box every single word to the um
functional units that get processed no
we batch them up in one vector like um
putting them on a pallet and then a
forklift comes and Carries them to the
functional unit where they get processed
and then often the forklift Waits at the
other end gets the result and puts it
back into Vector
registers so um why I care about Vector
Computing um well from application Point
kind of view
um there is scaler code and there is
parallel code sometimes they are so
tightly integrated that you can't easily
ship off the parallel code to the GPU or
something because um the latency in
overhead would not make worth it so um a
tight integration can be beneficial also
um we might be experiencing a Resurgence
of vector Computing through whisk V and
the Whisk V has this uh whisk V Vector
extension which is um slowly finalizing
and uh getting chips made for it but um
with a pre-release version of it
0.7.1 there is already um Hardware out
there which implements this um sadly out
outdated and in principle forever
obscure um subvariant of risk V which is
incompatible with the main thing one um
example of such a board is the um milky
Wii Duo which is a dual core um whis Wii
processor which has a small Vector unit
satisfying the spec mentioned on the
slide so I'm not here to talk about that
tiny Vector computer I'm talking about
my Vector super computer which in this
case um is two NEC Vector engines 10c
this is like the most bottom of the
barrel model this is a workstation model
with um blower fan not like water cooled
or something fancy and um as such uh I
have two of them connected over pcie to
an um AMD Fred Ripper 1950x because I
needed the PCI lanes and yeah I will get
to that it will not be funny and just
because something fits doesn't mean you
should plug into it so if you happen to
end up with a neck Vector engine be
aware that even though your PC8 pin from
the GPU fits inside that Port it doesn't
belong there it fits but it shouldn't
like with consumer electronics you
should expect or like with elect in
general if there's like some plug and
some cable and they don't go they
shouldn't go together then they
shouldn't fit together but in this case
for whatever oversight um yeah um the
neck VOR engine uses an EPS 12vt
um connector as it is more commonly
known as the8 pin CPU connector and like
has the same rating should you not have
a spare uh 8 Pin CPU connector from your
um uh power supply um Nvidia has the
same problem with their or the same
circumstance with their workstation
cards where for example the A4000 also
has the same connector and there is an
um connector made for
such Nvidia cards where you can put in
two eight8 pin PCI cards which then
together
um uh are available as one EPS 12 volt
and Supply sufficient voltage to the uh
to the 300 wat which can be drawn via
the EPS 12vt connector so um we talked
about pallets as like
um things which we use to transport
things and if we like compare uh what is
a vector computer what does it mean to
build something as a vector computer
then we should like also consider the
size um if you recently upgraded your
desktop CPU um you might have done the
switch from AVX 52 from AVX 256 to AVX
512 so you now have 512 bit wide vectors
which is completely utterly draft by a
single Vector from the neck Vector
engine which is
16,34 bits wide it might be more
convenient to think of it as 250 56 46
bit
Words As for um not only are the
registers on the vector engine bigger
it's also twice as many of them so
um it's a very different design but
um that's what I'm here for to tell you
about so uh how does it work internally
are there like 256 uh arithmetic
arithmetic logic units in parallel no
not quite um they
um internally work on groups of 32 46
bit words and uh there are five
functional units and three are for
flirting Point numbers the first two
mostly do like fuse multiply add and
that kind of operations which occur in
linear
algebra um while the uh third FPU is
fancy I would call it and has uh in
Hardware instructions for square roots
and Division
and other fun stuff like that while alus
um operate on integers and the
um and as logical operations on the bits
which make up those integers like
Notting a number this happens on the L
now
um the neck Vector Engine with like its
um reordering capability and the
pipelining capability um can sustain all
those units um keep them fat and get
results back um this uh in the end has
to be backed of course with 46
6,348 bit register so one mbit of
registers alone per core but um a vector
computer is not just Vector units
there's also always a Scala part with it
and well this has has its own sets of
caches and in the end before the data
goes out through the hbm there's the
last level cach which is shared and um
thus also provides synchronization
between those
two so um I want to start with a simple
instruction um which um nowadays is
pretty common and not spectacular but um
has sparked wild conspiracy theories in
the past as NSA instruction when they
suddenly appeared in AMD CPUs without
much fanfare the idea is um you have
some number and you just count how many
ones are in there this is useful for
very cryptographic applications and I
want to like um show you some Snippets
from the um neck iser guide so um you
can like get a grass powered looks and
in this case we have two Scala registers
and we um the result of SX will be the
population count of Sy
y um the link down below is a was a link
to a um longer discussion of the like
NSA thing and but that's not enough and
the Nector neck Vector engine has also
support for a really uncommon type to
scene Hardware namely quadruple
Precision arithmetic so floating Point
numbers which are 128 bits wide that's
only in the Scala part not in the vector
part and it works by basically stringing
two 46- bit registers together and then
just you can add multiply subtract no
division but it shows like this is a
kind of hard operation you usually don't
have for this data type then like um
Vector predication um but to introduce
this I first want to do like a introduce
a basic Vector operation so far we only
had Scala ones um if we look um at the
um functional description below we see a
for Loop um which iterates between zero
and VL minus one where what is real and
with a vector computer you you don't
always have to use the entire Vector
length
and you can say well I only want to
iterate over 180 things and then it will
smartly discard the rest and like do
things in Hardware to make um to not
waste as much
resources and this way it is really
convenient if you have a for Loop and
there's just some remainder remaining
well you just do one itation with that
and you don't have to like have special
taals which unroll in different ways all
these RS are gone then um we also have a
flag in this instruction funtion
um if a CS is one then just a single
Scala value will be used to like shift
all values to the
right while um if CF zero um instead a
different Vector will be queried and
every element will have its own shift
why have both um well um communicating
one number is a lot cheaper than
communicating 2, 256 numbers so I guess
from that standpoint it makes sense and
the I are
is designed to let you ex to let a
knowledgeable programmer or
knowledgeable compiler exploit such
information and then the interesting
part is this if with the vector mask if
the vector mask is not is not said
nothing happens and if it's that then
Rex is changed
and this comes from the idea that
sometimes doing a bit of extra work um
is quicker than um trying to um
eliminate all the extra work for example
like if we you were a bus driver you
wouldn't remove all the bus seats you
wouldn't be using you would just be
keeping the bus seats in place and drive
from A to B and if people don't use it
there's no need to put them outside to
save a few grams because that will be
more effort overall a similar analogy
applies here so those Vector masks don't
just um randomly emerge um you can
create them by like comparisons
operations that kind of stuff but you
can also do operations between Vector
masks this um if you're really You Want
To Be Clever about this might allow you
to implement some State machine where
you then can basically have 256
um uh parallel instances which all run
through the same state machine which
could be need for various pausing or
something like that application in
particular since the ne Vector engine
has an generally an enormous memory
bandwidth
then does someone know this
code um
do you yeah there's also colorful
language in there and this code comes
from Quake the also it's attributed the
hisory of Thea is longer but not
relevant there is some evil floating
Point trickery in there I'm not going to
explain
but how this is interesting is that the
neck Vector engine has exactly this
operation not exactly this
implementation but exactly this
operation in Hardware so it has an
inverse square root in Hardware why
would why would one need that well it's
designed for scientific applications and
sometimes there you want to um normalize
some data set or um it's also useful for
calculating like velocities in 3D space
and um yeah it has a square root
operation it um has a vision operation
why also produce a one over square root
well um we only have like one unit which
can do division and square square roots
and if we just produce one over square
root we can just use the normal multiply
unit and that way um we save Cycles on
the fancy FPU unit what is kind of
unsatisfying about this documentation is
um the Precision of this approximation
is system dependent so um if the last
Vector engine dies um we will never know
um I don't like that idea I tried to
like uh make some progress towards
documenting how it works for the 32 the
case that is easy because um there were
just
um well that fits in Ram basically 2 to
the 32 is not that much anymore but for
the 46 bit case I will need to be a bit
more clever and we also see here with CX
and CX2 that um it's possible to operate
on the lower half or the upper half as
single precis or both at the same time
or as one
64bit then um this is a instruction when
I saw it um I my reaction was what the
fun is that um if someone has a
suggestion um you're welcome but also
like the description doesn't yield
anything and um the person who has like
the most convincing idea after the
questions and talks to me um will get
like a a Milky we Duo small Vector
computer I talked
about but um I'm kind of puzzled by this
one
so a short summary um it can execute
scaler and Vector code and if you know
that the scale and Vector uh
instructions uh load and store from
different addresses you can also say
that like the vector unit can on certain
operations ignore potential dependencies
to like unlock more performance it's
primarily made for scientific um
computation and has a powerful and a
forgal Isa with very right vectors of
orthogonal in this context means there's
like a clear separation of concerns and
you can just looking at it get a pretty
good feeling what instruction does there
is no complicated decoding which needs
to be undone um which is kind of nice
and I have a bit more um it also has
like powerful Aus which can do
everything a desktop L you could wish
for from except like it can do it with a
lot of you a lot of stuff at the same
same time it also has a powerful FPU
what is not supported are Shuffle
operations except in the case where you
want to swap around 32 sized groups
because that's also what is used
internally and just the reordering logic
is exposed to the user here for the um
10 and 20 generation there is no support
for um 16bit um reduced Precision
numbers as are typical machine learning
with and the third generation this was
introduced and um if someone asks for
pricing and I think I got them quite for
a steal and I noticed that they were in
my opinion sold way under value
somewhere in China and I had a Chinese
childhood friend I asked to well get
them for me and using well
for uh 48 gab of hbm in total I think
alone that price will be worth more than
that then
um it's also recently turned out that um
the huge memory band with those Vector
engines have is really it's also really
suited for certain um uh machine
learning application in particular
inference um if we look at this sport
snapshot we see like six tiles around
the main tile these are like hbm Stacks
which is high band with memory and um
normally you only get like four with
with something usually
so um this later was also um someone
tried around with it and the story
behind this is actually interesting this
was kind of motivated by the live stream
where someone thought like Express AI
this person thought he owns Vector
engines himself and I later joined the
live stream well
um that um trying large language models
on a vector engine would be fun and well
then there was some collaboration trying
out some ideas and well it turns out
Lama do 2.c is just like a plain C
implementation of um
um large language mod just to understand
it and like it didn't run perfectly
right away but like when we remov the Q
sort because like we don't want
everything sorted we just want the top K
entries and um we switched the memory
orders of the matrices from column major
to row major and then suddenly and the
uh NE VOR engine which is new and mine
handily beats like an a100 in that
particular
workload so talking about um the
software which comes with it and where
it's stored uh on the neck Vector engine
itself there is as with any modern
electronic product firmware there's an
arm cor on it which does power
management fan control helps with Device
initiation the kernel drivers open
source and like I've successfully build
it for a different Linux version than
the S when the officially provides for
Redhead but I've not tried to build it
for different non wet head derived thing
there's a close Source binary in
userland which manage things from the
host there is there are compilers and
there are like typical scientific um uh
libraries like blast the fastest forer
transform in the west fre par stencil
codes and
MPI MPI is fun um just some other
binaries which are shipped with it
that's kind
of kind of fun stuff um just some thing
so about the compilers there is a
proprietary compiler by neck which
supports in x86 46 and the vector engine
as a Target it can't be open source I
asked then there are two um offical then
there are two lovm derived compilers
which are distributed which are both
based on horribly Out ofd lvm versions
um the reasons for that are kind of on
the slide um Upstream lvm supports the
neck Vector engine but it's not a build
Target by default um you can emit um all
the vector instructions if you want but
the vectorizes in those compilers aren't
quite strong enough to make good use of
those heavy Vector units then um yeah I
uh promise home you here is some 2048 on
a neck Vector engine I just took a
single C file um 2048 implementation I
ran that through the neck compiler I
used F trades which will become relevant
later then I ran the program I played a
bit I got I got a score not a good one
actually I just wanted a screenshot and
F TR is where it gets interesting
because F TR is a tool which um reads
out performance counters and makes them
available to us and using that we can
use that as guidance to what functions
are potentially holding us back in terms
of performance in this case um often
interesting um columns are the uh vop
ratio and the average Vector length in
the main function we have an average
Vector length of 16 what is six what
does 16 have to do with 2048 yeah it's
the board initialization it detected
that for Loop it vectorize it follow it
didn't find anything else to vectorize
so it ended up with an average length of
16 and in a drawboard it found something
else small to vectorize but for example
the pushing blocks around based on um
pressing a key it couldn't vectorize
that and um this can just this is an
application you wouldn't want to run on
the vector engine because it doesn't
make good use of its resources and then
it's just like a slow scaler
CPU um now uh soon audience
participation htop is a command line
task manager on Linux so um if you
compile htop and its dependencies such
as en curses and so on uh with the
vector engine as a Target and run it uh
what do you think will happen uh race
your hand if you think a it won't
compile
o1 okay um B it runs on the vector host
and shows Vector engine
processes C it runs on the vector engine
and shows Vector engine
processes D it runs on the vector engine
and shows Vector host
processes okay and I didn't understand
the question but want to rise my arm
anyway you're welcome thank you for your
participation it's
D um so what happens here um
the uh Vector engine like the
information htop gains about a system
often is not like through system calls
directly but instead by accessing the
file system those file paths are kept
from the x86 version so it looks in the
same place as where it would find the
x86 system information a few data things
are obviously corrupted I don't think
like this with the terabytes of virtual
memory makes that much sense and me
being over 100% there's like something
odd going on there but um overall um I
can show that it runs on the vector
engine but it displays data from the
host in this kind of corrupted way so is
it is it surprise you take some Linux
cc++ for an app compil it from source
and it runs um and then you just slow
tweak the slowest part and it works and
then suddenly you have high performance
well there's some difference
and they are elaborated somewhere else
and the support for other programming
language is
thin oh then um Vios uh is what fakes a
um Linux user land towards the process
is running on the um Vector engine and
certain system calls like Moc are
handled directly on the while all others
are passed through the um Vector host
um it's best to think of the vector
engine as a co-processor because it
doesn't have its own timing Source what
actually happens to switch processes is
that the uh Vector host the in Intel or
AMD CPU holds the PCI device swaps out
uh the content of the processes State
over PCI and kicks it again because it's
kind of an expensive operation uh the
scheduling interval was decided to be a
second so it's also kind of funky
watching how processes behave on there
because they always stick for a
second then um for
there we had like situation where we
have a binary running on just on the
vector engine like with 2048 top there's
also a possibility to take Cuda like
codes and use a replacement called Vader
and have the um main CPU offload some
Tas to the vector engine but this is the
first time I've seen this in
architecture you can actually do the
opposite you can have an application
running on the vector engine which then
hands off certain scaler intensive
workloads to the CPU and say you deal
with that and I I keep crunching my big
numbers but I'm in charge um and that's
like kind of blew my mind then um it was
not everything Ry I had problems getting
it started and um so I decided to dig in
the binary a bit like um I didn't find
anything with 16 I had problems with PC
I thought like maybe it was a 16 versus
eight Lane problem it was a problem but
I didn't find anything in the binary so
I looked like for version three and
version two maybe I found something else
when I looked for two I found something
else I found this snippet which I kind
of likely
annotated and if we change the um the
how de how deep to sege determines um
how many options we see when we uh
interact with the command line and by
just changing that one to a two we
suddenly left hand was before right hand
we unlock interesting new operations
like flash weed right dump but uh a lot
of interesting iio but well the most
interesting thing is the signature
because we can now um decrypt and
re-encrypt and the um Pro the scripts
the um proprietary binary was executing
so it turns out there were instead of
eight there were 10 cores on a vector
engine and it might be able to enable
more than one of them or tweak with the
HPM uh megahertz rate
so yeah uh Vector engine
has a bunch of um properties um you
would want from an HPC cluster it I
think
um okay let me just go quickly through
that um you can split a single Vector
engine in two separate ones by
separating the central range stem
basically and eliminating some numer
problems and it's connected by
internally fast Network which will be
over PCI the file system is shared as in
it's shared with the horse because
everything happens through the horse
users have no access to the compute
resources themselves this also true with
the vector engine because you can't s h
into the vector engine you can only
assess H into a CPU into a vector host
with has a vector
engine execution we have job scaling is
possible slurm supports the vector
engine Kinder
um and like vectorization and multinode
parallelism also unlocks a lot of
performance with the ne Vector engine so
it can be like a little toy box of all
the problems and all the fun you can
have with HBC in one form factor drawing
less than 250 Watts if really pushed so
there are multiple users but I actually
forgot to put one down the one closest
by is uh the uh claan Centrum a climate
research signs thing um just a few
hundred meters I think in that direction
I'm not sure and they also have neck
Vector engine so
um that are the things I know and not
only those prestigious in not only those
prestigious institutions but also me um
if you send me your uh preferred
username and public key I can give you
access to my Vector engines you can play
around with it
and yeah um I'm open to take
[Applause]
[Music]
questions so if you have questions
please uh go to the microphones in the
middle of the room
therefore
um signal Angel do we have questions
from the
internet uh yes we have one
question that is how did you get your
hands on this card well um I uh first
heard of it um through like Julia con
where someone from neck basically told
us the hardware exist and someone tried
to like make Julia work on it a little
and then I just um noticed at um in my
feet some uh Japanese people I followed
suddenly started posting about neck
Vector engines being used being
available on the used market and then um
I asked where and then I learned well in
China and I happen to have a Chinese
friend and well then uh he bought them
for me I sent him a German Christmas
food as a thing in return for the effort
and um we kind of had a deal with that
and I had the first one uh last year
around this time and the second one
arrived recently so I hadn't had much
time to do firm VOR mentation because I
couldn't bear the thought of having no
Vector engines to have access to but no
that know that I have two there are a
bunch more
options okay uh then microne one yeah so
um thank you first of all for the great
presentation um so I want to ask you
about this um example that you showed
about htop doc right so by default what
I
that um GCC does not support it you have
a proprietary compiler right and an and
lvm based compilers and there is um some
work towards GCC but it seems pretty
abandoned it looks like neck decided to
switch to lvm at some point but there is
some reppo and I'm not sure what state
it is in okay so um if uh this or
something like this probably this five
right um comes up with this kind of a
vector in
um the entire user land needs to be
compiled to take um to use this
acceleration like what happens on the
GPU uh there's a Cuda driver that that
is involved right it has an nvcc
compiler and it sends everything to the
accelerator GPU to get things
accelerated not only that's one option
the other option is that you can just
take an application and uh just execute
it as an application on the neck Vector
engine and the complete control flow is
on the neck Vector engine okay yeah yeah
but then the entire user land has to be
compiled for no discussions uh is that
correct is that um it's a different
instruction architecture of course okay
thank you so microphone
s the confusion Vector instruction you
mentioned I think it's for ffts where
you basically have uh the elements of a
complex number interleaved so you have a
vector of complex numbers with both the
uh real and the imaginary part
interleaved in the same Vector register
and you want to do the basically the
butterfly operation and there you do the
same operation just with different signs
on pairs of elements that's why you have
that instruction do you also have a
question uh you just wanted to know that
where it does that and the question
would be uh how how portable to
different Linux distributions are the
host tools are you required to run
Center as a rard I've have not
experimented with anything else I heard
people trying it with Debian but I'm not
sure whe I gave up because of struggle
or just because of lack of motivation
and
interest I haven't heard that it's hard
but I haven't heard that it's easy
either signal Angel do you have more
questions uh so far I don't see any more
questions um then microphone one hi um
um thank you for your talk um I love
these cards I um we played around with
them about 5 years ago we we had a nice
uh H HPC class where we compared the
upcoming uh Intel many cor offerings
with next Vector offering and we
basically found that the many core
systems they're [ __ ] and uh the next
systems are the [ __ ] so that was pretty
fun um one thing we had back then was
back then they only had the NCC compiler
and it was kind of me so what's your
experience with these three compilers uh
do you have any results on how well the
other ones perform in comparison uh I
heard that the uh rvv uh was really
Advanced for its time uh the region
vectorizer one uh
I've and I'm not I've have not read
enough into it to like be qualified
enough to make a statement but
definitely um worth a try and the next
is also okay yeah and so about about
your instruction it looks a bit like a
Horner them yeah I think I want to take
another question we can chat later of
course yeah I what I wanted to give my
guess for your instruction but you
can okay I take guesses later we just
two
questions hello yeah um please no
discussions at the microphone we have uh
some more cues please uh make your
question short and precisely microphone
three please yeah thanks for your talk
um I enjoyed it and I would like to ask
um you mentioned that the NEX also had
some kind of rather deep Pipeline and
I'm not sure how exactly it works but
could you elaborate a little bit how it
compares to a modern x86 pipeline so is
there a proper kind of brunch prediction
that usually gpus don't have uh not only
brunch prediction but as far as I know
also um mild to aggressive speculation
about when vectors are finished and uh
then in validation of results so uh the
uh job of
the scaler unit is basically to keep the
scaler code out out of the way so the
vector unit can run and for that
also I
I I'm not qualified enough to like
evaluate that but yeah
thanks Microsoft microphone microphone
one please
uh could we go back to the uh very
beginning with the vector word size
Vector word size because you uh
mentioned it's 16 348 and uh it do I
have do I have the last two digits
switch up uh I I don't think so it looks
right it's uh doesn't look like precise
16k is it intentional or uh it is
shortened uh internally for some reason
it's um it should be 256 64 the rest you
can um math math out yourself so yeah so
it probably should be
16384 you're thank you for your
question um yeah okay we have um um
signal Angel do there are no questions
more questions from the internet
Microsoft 3 please sir you mentioned
that the vector CPU is out of order
capable um to some degree yes do you
know um basically if you use uh
predication does it really basically uh
take advantage of the out of order
capability if you have a predicated SCA
GA instruction to really process and
chain partial results as soon as they
become available and process the
elements inside a vector out of order or
is it in order per vector or something
I'm not sure um with ga and get
generally reduction operations I don't
know and other thing how high are the
contact switching costs if you want to
switch be between multiple vectorized
application do you have to basically
dump the whole uh accelerator memory or
do you have any kind of ability to
partition the card and quickly switch
between um there are eight cores or
alternatively two times four cores and
those cores are cores and if they want a
context with switched the state of that
particular core needs to be dumped to a
main memory of the CPU and something
else needs to be loaded in over
PCI okay final question Microsoft
One hey great talk man um my question is
how easy is it to program these devices
like sometimes I see people you know
using manual uh simd instructions for
x86 and I'm wondering do they have to
get really low level programming of
these what I can say uh that um people
are able to like take some code they
kind of know like someone once uh came
uh try to run a uh mandal bro plotter it
took like a few changes like something
about a while loop had to be changed and
then after some more changes it kind of
vectorized decently and what's what I
kind of like about the vector engine is
that even like the uh there are no no no
really obscure instructions you can't
guess because the shuffle pattern is
somehow strange it's it really follows
the idea that like each piece of data
stays in its Lane and like the only
thing where this breaks is this you can
Shuffle 32 block 32 46- bit word blocks
but other than that it's really
straightforward to predict what it does
cool and to search it for functionality
you might want yeah
thanks okay then please applaud this
great speaker Yan
toia
[Applause]
[Music]
sh