Dear viewer, these subtitles were generated by a machine via the service Trint and therefore are (very) buggy. If you are capable, please help us to create good quality subtitles: https://c3subtitles.de/talk/893 Thanks!
David Lutfullah Twilight and the title,
there is a very long sentence because
it's on the prospects and challenges of
winter and climate modeling
at conviction, resolving resolution,
to my opinion, it's
trying to make complex
thoughts and views
a little bit more easier.
Do you understand?
Please give David a warm welcome.
Applause And let's
kick off.
All right.
Welcome from my side.
Whoa. It's really terrifying
to stand here. So please excuse me.
I'm extremely nervous, but you've
probably heard this a few times
now.
Before I begin, I would like to to thank
a lot of people. So first of all, all the
volunteers that make this conference
possible, it's been amazing so far.
I've seen so many interesting talks.
I've had so many interesting discussions.
And second of all, I would like to
thank my my coauthors for this talk.
They're my team.
I work with them on a daily basis.
But for this talk in particular, I would
like to thank Christopher
Welliver for unharnessed forked,
who contributed some slide, which allows
me allowed me to stay a
few days in the last week
with my family and start preparing
slides.
So who am I?
My name is David.
I'm an atmospheric scientist by training.
I currently work
in Zurich and they're in a group
which is interested in
the summer climate over Europe.
And I work on the interface between
atmospheric science or climate modeling,
computational science and computer
science.
So
this is basically the summary
slide from the first half
of the last talk.
So the climate system is warming
that is unequivocal and the human
influence on the climate system
is clear.
So now that we've established the climate
is warming in the last talk, we
it's time to go a bit beyond and
look what's coming in the future.
So we know the climate system is warming
and we're very interested in
how a future world
might look like.
But then at the same time, there's
this famous saying
making predictions is hard,
especially about the future.
And one way to do that is
by using numerical models.
So trying to put the
Earth's system into equation
and use climate
models to make projections.
But we we
make our models to answer
very specific questions or very
specific test, very specific hypotheses,
and we don't
want we don't want to create
the earth in a virtual world.
So this is kind of important
to remember. So all the models are wrong,
but some models are useful, especially if
you have the right questions.
So this is another summary slide from
from the last talks is the very famous
IPCC slide.
From the summary for policymakers and
essentially tells us that we have a
choice, depending
on the amount of carbon emissions,
we can either live in a world which
looks which has a temperature change
like the plot on the left, or
we can keep a business as usual
scenario and we can live in the world
that looks a bit more like the world
on the right.
So this is a plot of
surface temperature change and it
contains a lot of
very useful, robust features
that have been addressed
and investigated in the past.
For instance, we see that the continents
are warming faster than the ocean.
We can see that the poles are warming
faster than the tropics.
And everywhere we are, we have this these
strange bloodstock black
dots is where all
the models that contributed to this plot,
a green sign and where
the signal to noise is very high.
So another way to look at it
is by looking at the
global mean temperature change.
So on on the bottom panel,
you see by past observations
and in the big panel, we again
look at the scenarios.
So we have a business as usual scenario
on top and we have a mitigation scenario
on the on the bottom.
This would be roughly the two degree
world I showed you before.
And in the straight
red line, you can see the mean
change predicted by a
large number of models.
I think it's around 40 and
the color shading indicates
the range.
These models span.
So this is
this is kind of the uncertainty range
that our models project.
And a lot of effort has been made
in the past to understand this
range, to try to
see where this range comes from.
And of course,
we would like we would also like to to
reduce that uncertainty range so we
know a bit better where we are heading.
And in this talk,
I want to look at
some ideas, how we
can we think
we can reduce this range by improving
our climate models, because we think
that such a large part of
these uncertainties comes due is due
to uncertainties in the response
of clouds.
And if we improve
the representation of clouds,
we might be able to reduce these
uncertainties.
So kind of an outline, an outline.
I first would like to talk about clouds
and climate sensitivity.
Then I have to
give you a very brief introduction into
weather and climate models, how to work
in a nutshell.
Then I would like to talk a bit about new
climate models which are able to
mitigate to to
not to mitigate to
work around some of the uncertainties
that are in the current generation model
of models that I would like to talk a bit
about challenges in
computing, weather and climate.
So computing these new models.
And finally, I want to give a short
outlook on what we can expect in the
next in the next five years
or the next couple of years.
So let's begin with the clouds.
You remember that I told you that a lot
of the uncertainties comes from
the representation of clouds and clouds
can react to warming in
the large in various ways.
And here I've selected a few
so the clouds can either contribute
to the warming by that
would be a positive feedback.
So as the temperature gets warmer, their
contribution makes the climate system
even warmer. But they can also have an
opposite effect and they can
mitigate that. They can moderate the
warming. That doesn't mean at all that
the climate will get cooler,
but that the increase will be
a little bit smaller.
So, for instance,
in the future world, we could have more
high clouds and that would then
contribute to a positive
feedback. We could have more
we could have more low clouds, which
would have the opposite effect
because the clouds
could get higher, which would again have
a positive feedback.
They could get less icy and
more watery, which would have a negative
feedback, or they could
change their position.
For instance, if they moved closer to
towards the poles, this could they can
have a positive feedback.
So you can see that we have a whole
slew of ways the clouds can
react. And here I've chosen just a few.
So there are more of them.
And our problem in
the current generation of climate models
is that clouds are not
explicitly resolved, but we
parametrized them.
So it means we use physically
based semi empirical models
to represent them in models and
climate models and the way to illustrate
this, this so we have
we have a model world and we can change
the world.
And here and one way you can
change it is you can try to simplify
it, to make it easier to understand
and hear what the pollsters have done.
They've stripped the earth of all the
continents. So now you have only a nurse
consisting of water and an ocean.
But nevertheless, these planets that
features that are very similar to our
climate system.
So we can see
there is an intertropical convergence
zone, intertropical convergence zone.
There are hardly cells there, extra
tropical cyclones, but in
essence that the system you have to
investigate is much easier to understand.
And now you have this this idealized
world and you can run it for a couple of
years and then you can
burn the entire system by four
degrees and it will be up here and you
can run it again for a couple of years.
And then you can take the difference
between these two simulations.
And this is what is plotted here and
is plotted for four to four different
models.
And you can see that these models for
different models here and in the
vertical, they react very
differently. So the change in cloud
radiative affects, the patterns look
different and the change in precipitation
also looks different.
So and
we are our hypothesis
is that these differences
are mostly because clouds
are not represented explicitly and
that the parameters involved
here have a substantial amount
of uncertainty.
So to to to explain quickly
what parameterization is, I
want to give a very short introduction
into what climate models are.
So what does a climate model do?
It has two important jobs.
The first job is it distributes
heat horizontally.
So we have differential heating
around the
tropics that there is warmer than at the
poles. And the first job is
to kind of mix these two states
and the way that this
is the following.
So this is a High-Resolution
simulation at 13 kilometers,
which was the animation is
by by Oliver Stepbrother's.
And you can this is basically now how
a climate model looks like from the top.
So this is not a satellite picture.
This is entirely virtual world.
And you can see that we have many of the
features.
So here, for instance, the tropica, we
have that the strong convection and
the defenders, the deep thunderstorms
going on here in the tropics.
We have that extra tropical cyclones so
that the low pressure system coming
towards towards Europe, you many
of you know this very well.
At some point we will see tropical
storms.
So here would
be a tropical storm.
And it is a lot kind of
these these eddies then mix
that that the gradient
between the tropics and the poles.
So this is one job.
The other job is to mix in the vertical,
you know, just a similar part from the
last talk.
So in a greenhouse atmosphere,
the atmosphere warms
your surface and somehow
a lot of the energy from the surface has
to get back in the atmosphere.
And this usually this goes
through the
water phase, so through clouds.
But still these these clouds
are very badly represented.
So we have one high-Strung
heat energy, mass transport, which has
been represented very well since the 70s.
So these are Manop and it's my karinski
and these people that have done an
awesome job.
But on the vertical transport,
we're still a bit unsure.
So some equations
or the climate model does, it basically
solves a number of a couple
differential equations so it doesn't
matter too much what this is.
But here on topic, for instance, see
the major Stokes equations, some might
recognize this below here.
So it's a creation of state or the ideal
gas law.
And this is from the atmospheric part of
of a climate model. This is the climate
model, Kozmo, which I'm using.
It is, among other things, also used
by the German Weather Service to make
the weather prediction
and now unfortunately
so we want to solve these these
these equations so we get the nice
motions you saw in the animation.
Right. And sadly, we
don't know the analytic solution to these
equations.
So our only way is to solve them
numerically using numerical methods
on a grid.
And this is very, very expensive.
So we have a numerical mesh
surrounding the Earth in three dimensions
and we solve the set of five equations
at each grid point.
And then once we have the we have
we solved the equations everywhere.
We have a new state.
And from then from that, we can then
progress in time. So we do kind of the
classical time stepping approach.
So now, depending on the computational
power you have available, you can vary
the number of grid points you throw at
the problem, or you can vary the
resolution of the grid.
And now you will have processes which
will be very well represented on that
grid. And you have processes which
are either not representable or
which have a length scale that is too
small to be represented on that grid.
So typical global climate simulations
today have a grid
spacing between 125 kilometers.
So imagine you have to respect
grid points here and you have a process
that falls right in between that's
then hard to capture by
the grid.
So we have results processes
and we have so-called separate scale
processes.
And from there we have a lot.
So we have, for instance,
chemistry, cloud, micro
physics, radiation, sensible,
latent heat fluxes, which are very hard
to represent on the grid.
And then we have some processes which
we add as new
model modules of reference
and surface model and ocean model, which
also has a similar kind
of set of equations.
And then we have a lot of process to
start halfling scales which are too small
to be represented on the grid, which is,
for instance, here, the deep convection
at the shallow convection.
So the the big thunderstorms
in the tropics or even in the extra
tropics, a next step would like a
bit to look at at that
in a bit more detail.
So here we have a very, very
wonderful photograph
of a thunderstorm over
Lake Constance.
This was in April and 2006.
And it's a very beautiful image because
we can see a lot of the processes
that are going on.
So, for instance, oops, here
we have Dopp and Downdrafts
region. So Verda, that the
air parcels rise very fast
with up to 10 meters per second
over here we have what we what we know
as the Anvil region.
So there's a lot of ice
in there. And here directly below,
you see something very interesting,
namely kind of the the hydrometer are
falling out of the cloud and
then forming these discussed fronts
near the ground. And the fact where we
actually can see them in this
image is because first, this case here,
they consist mostly of ice particles.
Normally the rain and the ice particles
makes this makes them a capturable
baiba camera.
So now imagine we want to we want
to resolve this in a model.
So we overlay grid.
And the grid now has a grid spacing of 10
kilometers.
So we have information at these grid
points. Right.
And at this grid points.
And now what the physical
parameterization tries to do, it
tries to capture all the processes
that are going here on here in between.
And this is an extremely difficult task.
And it has been worked on since
the 70s.
And some of the most brilliant scientists
I know, I have worked on
this problem and we still
even today have to have the uncertainties
that have shown you.
So we are proposing to take another
route, maybe
a simpler route.
We propose to simply increase
the resolution of the grid
because now we have
a few points
representing the cloud and we actually
can start to resolve the motions
associated with these processes.
We can go even a bit further and then
very well capture many of the features
of this cloud.
So we've done this.
We as I said, we use the Kozmo weather
and climate model and we've done this
for a simulation
over Europe.
So we have done three
simulations, one at 50 kilometer,
one a 12 kilometer and one to two km.
And we were able to do this two kilometer
simulation because we use a very special
version of Kozmo,
which is a version which is
able to make use of the capability
of GPU accelerators.
This was an effort led by
by Oliver Futer.
And we used
this model to to run a simulation which
has about 1500 times, 50 times 60
grid points.
And we simulated the period of
about 10 years on 144
notes. We can run one year of
simulation in about two and a half
days or Feltl time because
of this GPU prototype.
So I can now show you a lot of
statistics, but this is boring.
So I'll just start I'll show you a few
cases from this. So some of you might
remember the storm.
I was Kyril and it
affected was a very strong wind storm
that affected Germany in January
2007.
And in this representation here,
the white shading indicates clouds.
The color shading indicates
precipitation with the blue colors having
very light, rather light precipitation
and the red colors having a bit
stronger precipitation.
And just to orient yourself that
the wind basically blows along
these these white lines.
And now at this point in time,
the storm is located here over northern
Germany and it exposes all the typical
features of tropical storms,
namely, here is a very nice warm
front with precipitation along the front.
And here we can see a lot of
cold front with a bit of precipitation
along the cold front. So we have usually
precipitation up to five
meters per hour.
Now, we can increase the resolution
of this model. And what you can see is,
first of all, we get a lot more detail,
but then we get also a bit more
precipitation here along the
cold front.
But kind of qualitatively, the
two simulations still look a bit
the same.
Now we're trying to disable
this convection, this parameterization of
deep convection and increase
the the grid spacing of
the model to about two kilometers.
And what you can see is that the picture
changes quite dramatically.
So we have physical features going
on, which we couldn't identify before.
For instance, here you can see these very
narrow bands or rather intense
precipitation. So these are cold fronts
and rain bands.
And if you look closely, it's
really fascinating.
So the wind blows here along
these white lines and the
rain bands are actually not perpendicular
to them, but they are they are
oriented in an oblique angle.
And you can also see that they're a bit
broken up.
And if you compare this, for instance, to
this radar image of which
was captured of an extra
tropical cyclone over the Pacific,
you can see that now.
Actually, these models start
to look like the observations.
So if you compare again, these two,
this looks like a lot more like two
observations.
OK, I'm switching seasons
going to the summer now,
the same kind of plot.
And we're looking now at the day
in July 2006.
So we have thunderstorms and rain showers
in summer.
And on the left side, you can
again see the 12 kilometer simulation.
And on the right side, you can see the
two kilometer simulation where we treat
the thunderstorms explicitly by
the equations of motion.
And on the left side, all the
rain that you're seeing comes out of
these parameterization.
So if you, for instance, now focus a bit
on the left side and you focus on the
precipitation, you can see
that it covers a very wide area
and that we have mostly greenish
and bluish colors appearing.
And if we watch closely.
You can see that it peaks around
noon, so now around 12
o'clock.
Now let's switch to the right side.
First, animation has now ended.
We can actually now identify
the individual bubbles so we can
see the individual thunderstorms
in the simulation because they're treated
explicitly and
some of the thunderstorms,
they have very small areas
of very intense precipitation.
And this is a very
different physical behavior of these
simulations.
And if you look when the
rain peaks, you can see that it peaks
right about now.
So in the afternoon and if you think back
when we experience
we thunderstorms in
summer, it's usually in the late
afternoon. It's usually in the beginning
of the night.
So now the timing of this
of the precipitation
in these simulations agree is much
better.
So as a summary for
for the first part, we want to
switch off the parameterization of deep
convection.
Because then we can formulate our
models much closer to physical first
principles, this
yields an improved representation of
summer convection and precipitation
extremes.
Has been very well investigated.
So here you can see a couple of papers on
that.
But the downside is these models
are computationally extremely
expensive.
In fact, if you want to double the
resolution of a climate
model, you need about a factor eight
in computational effort.
So to go from the 10 kilometer
range to the 10 kilometer range,
we need to somehow find a
factor, 10000 in computational
effort.
And I'd like to
show a bit what we've tried
in this direction.
So computing
climate models is extremely expensive.
And the way we do it is we use
what we call supercomputer.
So what makes the supercomputer super
is the network.
So we have a bunch of nodes.
We have a bunch of sockets that are
occupied by CPU's or CPU's
or whatever.
But the important part is that they're
networked together with a very
strong and fast network.
And then we can
slice our problem into smaller
chunks and distributed to the
individual what we call them nodes.
So. So, yeah, CPU's are GPS
and this is the computer I'm using.
It's called State.
It's located in Lugano
at the supercomputing center there,
and it consists of about 5300
hybrid nodes.
So each node contains an internal
hospital, multicore CPU and an Invidia
Tesla GPU.
And it is one of the larger machines
of this world. So some might know the
top 500 lists or currently is number
three. And what's more important
is that it is also one of the greenest
computers in the world.
So if you measure, for instance, CO2
footprint, so
it is a huge it is a
use of machines.
Are you to gain access to basically write
the proposal and which then
gets sent into review.
And if the judges like you, you get some
compute time on it.
But the important part is the thing is
that 90 percent of the compute power
of this machine comes from hybrid nodes.
So they contain CPU's and GPS.
And we think that the GPS are in
chips, which are very suited to
compute weather and climate models.
And this is because on graphics card,
so graphics processing units or GPU
accelerators, whatever you like to call
them, they're substantially parallel
and their throughput oriented.
But more importantly, they provide a
much higher memory bandwidth
and they're out and they're able to
highlight some of the memory access
latency between a lot of threats.
And this is important for Bretheren
climate models because in fact, many
of the operations in these models
are limited by memory bandwidth and
not by the available fops.
So not by the available floating point
operations.
Just to quickly explain what I mean by
memory bandwidth, we look
at one of these operations we do
in the most expensive parts
of of the climate models and the dynamic
dynamical core and in their many
operations are characterized by
the fact they need information from
their neighbors.
So to update the grid point here,
the Orange Grid Point, we need
data from
the neighbor on the left and the neighbor
on the right for for this easy case here.
And if we write it down, you can see here
that we have a of eight plus
one and then a fire and then a of
minus one.
And so we have to transfer a lot
of data to update the single breakpoint.
And the way to express it is
the Artomatic intensity.
So it says how many
flops we can do or how many
data or how much
data we need to do a flop.
And the easiest way to assess how much is
by simply counting.
So cannot count all the floating point
operations here.
One, two, three, four,
five. So we have five flops
and then we count all the data accesses,
which is one, two, three, four,
five, six, seven.
OK, so we're seven times eight by
10 single precision, so we have about
we do about zero point one flops
per transfer byte.
Now we want to know whether memory or
flops ltds. And for that we use
a very simple analytical model which
was established by Williams.
That's all from from Berkeley.
It's very insightful because it
essentially says that
Detainable Dobel cloned floating
point performance can be separated
into two regimes.
So one is limited by flops
and the other one is limited by memory
bandwidth.
So if MATTICK intensity
is above one, if we do more flops
than per byte, we are
in this regimes here and we are limited
by the by the peak a
floating point performance.
Or if we do, if the
automatic intensity is lower here
in the memory bandwidth bound regime
and we can go back and look at the zero
point one fluffs probate and we find
that we are here on the left side
of the curve.
And this is typical for for
the Stancil operations, which we find in
the dynamical course of weather and
climate models.
Now, if we increase the memory bandwidth
available, of course, our
time to solution will go down.
And this is why we think GPS
are good for our models.
So we ported our entire code.
And the entire thing is very well
documented here in this paper
by Olly, and so Kozmo is
about 300000 lines of Fortran code,
the dynamics so that the part treating
the equations that I've shown you
have rewritten C++ and
we use a domain
specific language that some
of us wrote called Stella.
Stella is nice because it abstracts
the hardware architectures from the
operations. So we have backhands now
for multicore and
written Kouda
and it allows this very low level
aggressive performance optimizations.
Well, so for instance, changing the loop
order while still maintaining a higher
syntax that the same syntax
at a higher level of the code,
the physics reports using compiler
directives. I was personally more
involved in this part.
But the essential thing is that after
about 10 person years of work,
we had the entire time stepping algorithm
ported and validated.
And this now avoids expensive data
movements between CPU and GPU each
time step.
So what did we gain from using
this new architecture?
And one way to look at it is
through a strong scaling
experiment.
So we take a given
experiment, a given problem,
which is in our case, just a given domain
size, and then we throw
an increasing amount of resources
at the problem.
And what we expect is the time
to solution to go down
proportionally with the number of
nodes we throw at the problem.
So this is what we show here.
So on the Y axis, you have time for
TimeStep, which is a measure four times
to solution.
And on the X axis, you have
when you go to the
you're right, we decrease
the amount of
the amount of nodes or we increase
the amount of grid points we compute per
node. So again, here we can see
two regimes here on the on
on your right, you can see the
linear scaling regime.
So here
the time to solution decreases about
proportionally with with the
resources you throw at the problem.
But then down here we see a saturation
regime. And this can be mostly
understood, I think, by by little
law for those who are interested.
So now what do we gain?
So here at the point where where we think
for optimal we gain about the
factor six in time to solution
or what's even more important for us,
we gain about a factor eight in
resources.
So now we have a very nice and efficient
model.
And the question is now,
how far can we push this?
I've shown you regional simulations
over Europe, but actually
the problem of climate sensitivity.
So the uncertainty range I showed
you at the beginning is a global
phenomenon and we wanted to know
how far we can get.
So we now do the other part.
We do so-called weak scaling
of the problem.
By scaling, we mean that we
vary the number of
nodes now.
Let me begin again be very the size
of our computational domain and
the very the compute the nodes
we throw at the problem proportionally.
So what we expect is the time to solution
to be constant.
And this is not a given, but
in essence, it means that for for
this experiment, theoretically, where we
simulated
over the Alpine arc, we use about 10
nodes for a European domain.
I showed you before.
We use about 100 nodes.
And the question now is, will this entire
model scale to the planet?
So that would be something around
5000 votes
and now the question we want
to answer is first, does it scale?
Second, what's the time to solution?
So are we there?
And we want to establish a baseline at
one kilometer. And finally, we want to
assess the efficiency of the
Cosmo model.
So we did this entire exercise
and we find now we again here
have the time per timestep on the
Y-axis, which is a measure for
the time to solution.
And here we have the number
of notes to the entire size of the
machine, which is about 5000 votes.
And we can see that the time to solution
actually stays constant.
And that's what we expected due to
this to the numerics that
are in the dynamical core.
So, OK, we now have
a model that can scale at
very high resolutions to two global
domains and we can do some experiments
with it. For instance, we can think we
can test whether we fast
enough to do global simulations.
So to this end, we
did an experiment of the idealized
American in a way, it's a very
famous test case for four
climate models.
It's by Yablonsky Williamson from
2006.
And we we
there we simulate the evolution
of American quave in the park, a grave.
And essentially you can think of it as
the extra tropical cyclones have showed
you before and what what you know, kind
of as autumn storms.
And because we have a regional model, we
now need to apply some tricks.
For instance, we glued in the west
east direction. We made the domain
periodic, so we glued together.
And we have a small
problem at the Poles, namely
because we have a regional model here.
You can see that the distance in the grid
gets very, very small.
So we get at pole.
Ultimately we get the singularity
and we have to address this.
So what we do is we simply
cut the domain at eight
degrees south north because
here we do a computational experiment,
but we never tell us.
We cover about ninety eight point
four percent of the surface of this is
simply the sign of
zero of 80.
And we then
used to set up to do a simulation, which
is about 10 days long.
And here we come to the.
Solution of of after
10 days. So here you can see this,
the barratry grave after 10
days, so we can see three,
a low pressure system following each
other. And then in one of the
low pressure systems we can see here,
this cold front or here,
the cold front, which gets wrapped around
the core of the low pressure system.
And if you look closely, we can see these
very funny things here.
These are small eddies which are
embedded into the mean flow.
So here we have a zoom
of all of the small
eddy, and this is
actually a new feature.
So this case has been run over and over
by hundreds of group, not hundreds, but
like a lot of groups, but only
at its grid spacings of
20, 20 or so
kilometers. Now that we can go to one
kilometer, we actually find new
phenomena and very established and
old cases.
OK, now coming to
the time to solution.
So when we do climate simulations,
you can do a lot of experiments.
But two very famous ones
or two that you do a lot is
the first one is you do a 30 year
long time science experiment so you
can, for instance, simulate the
last 30 years until today,
and then you can simulate 30 years
at the end of the century.
So, for instance, 2070
until 2100
and then compared.
So this is a very famous type of
experiment.
The other one would be kind of centry
scale simulations where you simulate
hundreds of years.
And of course, to do centry
scale simulations, you would tend to
need to be about 10 times faster.
So to do these 30 year long simulations,
you need to be able to simulate the
few months per day.
And if you now go to our benchmark's,
we can see that here, dizzier
point two, three NYPD.
So this is simulated years
per day.
So these are a few months and we can
reach that today at the KM, at
the grid spacing of about two kilometers.
When we scale to these to the
to the full machine at one
kilometer, we need about another factor,
five to seven to
get to this this window and
of course, to this century scale
simulations.
We need another factor, 10
to the century scale simulations
at one kilometer.
So we have a big challenge to go.
And of course,
these simulations come
at the cost and
we measured the power
they're required. So the energy in
megawatts of simulation year
and then extrapolated from there to the
30 year long simulation.
And you can see the difference that one
kilometer use about 18 gigawatt
hours.
And you can then using
the carbon intensity of the Swiss energy
mix, you can then convert it into
a CO2 two solution,
which would be for a one kilometer
simulation, would be about 3200
tons of CO2 equivalent.
Just to compare this
and we'll see an annual
footprint of a person
is between zero point one
and 10 tons
of CO2, depending where
you live on the earth and of course, on
your lifestyle.
So we have a big number here,
but we have to bring down
we have a lot a lot of ways.
How are we trying to do that?
So we can look at the energy mix,
we can look at the arithmetic
or the algorithms in the model.
We can look at the math.
We can look at compilers.
We can look at the computer
architectures. We can look at the chip
architectures.
So there are a lot of questions
still open and a lot of potential
to address these problems.
But nevertheless, we cannot also
say that we can do global simulations
with with two kilometer resolution.
So the nice animations I showed you,
they also apply.
But we wanted first we wanted to
establish a baseline and we
wanted to know how efficient we
are on today's computers.
And to this end, we propose
to use a new metric, which is called
the memory usage efficiency,
and it draws from two main ideas.
The first idea is that if you're limited
by memory bandwidth, of course, you
have to get as close as possible
to that upper bound data showed you in
the frame from the roofline model.
So the first term in this in this
metric is the measured bandwidth.
So what you measure during execution of
your model and then, you know, nominator
have that achievable bandwidth and we can
get that through a set of highly
optimized micro benchmarks.
Then the other point is that the other
ideas that you've maybe heard of the
expression floating point are free.
And this comes from the fact that
floating point operations on today's
architectures, they're actually much
cheaper than data movements.
So they're actually about a thousand
times cheaper in terms of time
and 100 times in terms of time
and about a thousand times in terms
of energy. So that's been measured a few
years ago by better controls.
So that the second idea is
that in order to reduce the energy
so this factor thousands here, we
have to reduce the overall
amount of data transfers
and we can simply count
the number of data transfers we do in
our model. But then comes the hard part.
We have to know the lower bound.
So how many
transfers are actually needed?
And to this end, Thorsten,
Carlos and Greg developed
a poor performance model.
It's a bit hard to explain
in two minutes, but in essence,
what it does is you have an operation
as the one I showed you before, and
then you look at what data
we require for that operation and
what do you write out?
And then you map these data
requirements, you map to your
to your to your architecture.
So, for instance, to do the different
cash levels.
And now then you can compare it
to what you what you actually do.
And that's the important part, is
this performance model doesn't only look
at one individual operation
at the time, but the important is if
you do an operation, it will have
predecessors and we will have followers
so they can share data and
they can share data.
And their performance model looks
at kind of the algorithm
or the implementation of the algorithm,
how it is written out, and then gives us
a lower bound of the required
data transfers.
So we obtain that for our model.
The number itself is a bit and not
telling, not informative,
but we can see that
we perform quite well.
So for for the first term.
So it means that the our implementation
fits the architecture
quite well, but we can do maybe
do a bit better with regard
to the memory bandwidth.
And now we hope that other groups will
take up this measure and compare so
we can learn from each other.
All right. That was mostly
it.
Here are my conclusions.
I hope that that was able to
show to you that we have some very
interesting, cool, fun
physics to to address
and that we have some very interesting
computational science problems to
solve if we want to attack these
big questions, such as
the climate sensitivity.
And there is a lot of potential for
improvement and there's a lot of
interesting science to be done.
And I hope maybe some of you
would be interested and talk to me
or tell me that everything I've done
is wrong and that you could do it,
all of it much more easily.
And that would be amazing.
So thanks a lot for your attention.
This is really fantastic.
You know how you do your work, I
can understand some people are quite
jealous about the computer power he's
using, isn't it?
Some questions, please.
Maybe one moment.
Sorry if you want to pose a question in
German, that's totally OK.
But I'm trying and translated into
English so everybody can understand.
So if you want to ask in German, go
ahead.
All right.
Hello again, thank you for the excellent
talk. Um, did you ever
try to dynamically adapt the
grid size like it has done in mechanical
engineering to make it more
precise or as little as possible?
In that case, we
ourselves didn't do it.
We had another task,
but it is a very active
research topic in my community, so
much less in climate, but a bit
more in weather
prediction, because there
you have to be much more
even more efficient than we are in
climate. And what you can, for instance,
do is let's say you're interested
in tropical cyclones and hurricanes.
And whenever you detect a cyclone,
a hurricane, you can dynamically
adjust your
your grid for climate.
This is a bit less interesting
in my personal opinion, because
the energy balance of the oceans is
mostly governed by these low
level, low level, very
shallow clouds.
And to resolve those clouds,
you need the grid spacing around
one kilometer and below.
So if you want to cover these clouds,
you need the extremely high resolution
everywhere anyway.
So maybe we should focus on that
first before we go into these
these refinements.
Thank you. Can I have a question from the
Web? Uh, there are no
questions.
OK, lots of others here.
Number two, please.
What observing the clouds
help with describing
them or simulating flying there with
aircraft or waves or something like that?
Um, yeah, we do that a lot.
Not not me personally.
There are a lot of metion campaigns,
the people flying through clouds with
aircrafts, people trying to model
each of the processes or how cloud
droplets form, how they grow,
how they collect each other.
And there are actually
two decades. We do we do observation
campaigns.
So one of the most interesting ones that
will happen, it's maybe right now
or in a few months is the Narval
to campaign is involved
is also the Max Planck Institute in
Hamburg. And they will do some very
interesting coupling.
They will do some very interesting
experiments. But at the same time, they
will also apply these High-Resolution
one kilometer or below models
at the same time and try to
to to leverage the knowledge gained from
both of these fields.
So, yeah, we do it.
And of course, we also use satellites.
There's a lot of observation of clouds,
but the problem is extremely hard.
Good No.7.
So I'm from a somewhat difficult,
different field. I used to do some
medical imaging stuff on GBW
and it was rather surprised when you said
that the memory bandwidth was
the most decisive thing about
your usage of Jupitus, which
is really unusual because usually
use that for for great computational
density per byte.
And so it was that mostly about, I
don't know, having access to
cheap, really fast memory at a large
scale, or have there been, like, attempts
to to develop a sixth that is
application specific
hardware for for your kind of
computations, climate modeling.
Um, well,
application. Well, why we use
them really depends on the
characteristics of your application.
And you remember you remember
that the roofline model.
Right. And you can see that
the increase in
kind of the difference between the two
lines was about as big for the
increase in memory bandwidth than
the increase in in peak performance.
But it's a very different view you have,
of course, because I guess you do a lot
of matrix matrix multiplies and D
operations, but they're actually useful
for both types of
problems now.
I mean, why we why
they were chosen, I don't know.
Well, I know, but I wasn't the person
that made the decision. This was this.
Yes. Yes. And they did a very thorough
analysis.
And but one reason is that
to build supercomputers, you
usually can leverage what is
there in the consumer market.
Right.
You can you can essentially leverage
what what what is already out
there and try to put it together
in a new way.
Are rolling your own chip.
I think I've seen that there is a talk
about that is very
expensive.
But nevertheless, there have been
attempts using, for instance, FPGA
to try and design codes
and chips.
And this is something that we will
certainly have to look at in the future.
But. Has not been done yet operationally,
so the people who I was talking to
is Donofrio at all from from Berkeley
and have some very interesting papers on
that.
Enough time questions, I see
some other people. Number three, please.
Thanks for the very interesting talk and
currently clouts already the
transfers modlin to the
end. Do you also plan to experiment
with 3D radiative transfer like
the 10 stream algorithm or something
similar?
Yeah, well, right now
are a radio difference, of course.
Is that the Delta two stream approach?
It is a bit. The question on
the process is you're interested in.
Right. If you're interested in a very
timely, high resolved, radiative
transfer question, of course,
want to go there.
But radiative transfer is
the other expensive part in
the model. So it's on one case.
It's the dynamics and then the
the other that the next one is radiative
transfer. And actually today we don't
run the radiative transfer
every time, but we currently run
it into two kilometer model about every
quarter of an hour.
And we think it's just
enough not to affect the results
of our simulations too much.
So first, we'd have to
to I would like to increase
the frequency we call that module before
going to two to
three D radiative transfer,
but.
Yeah, it's a trade off, and
currently we think we're
it's OK still, but if
you want to go to one meter, yes,
there are is a lot of work, for instance,
using also array
tracing and these ideas.
There's a lot of work going on in the
community.
OK. We still have ten minutes, I see
question on the on the Web.
Yes, yes.
Of the Fortran code you
were talking about has been around for
quite a few years.
What about the paralyzation code?
If the architecture changes?
OK,
so.
You basically you have a number of
ways you can approach this disappointing
problem, right, and this depends
a lot on the politics going
on in your community.
What is this community trying to be
and where does it try to go?
So, for instance, there is there is one
idea that in order
to support large,
large scientific codes, the different
architectures, you simply you have
kind of the master branch of your code
and you try to keep that as clean,
as nice as well as possible.
And then whenever a new architecture
comes around, you simply take that call
with your branch off.
You let a shitload of students on it
that report it, and then you have a new
new code, the other approaches.
But what we were trying to do,
so Kozmo, is what we call a community
or consortium model.
So it's about, I think, about 15
weather services around the world that
want to use it.
It's about more than 160 universities.
And I don't know how many individuals
under from the top of my head.
So they all have different machines.
They all have different architectures.
And we chose
a way which where we try
or all he did, he's the mastermind
the way that we have.
We trying to contain
the single syntax at a
high level. So we don't we want to make a
minimal amount of changes between
architectures.
So this is why they use
this this stellar they
developed a stellar library in order to
to make
low level optimizations
to the code.
And that's all.
Then we use the compiler directives,
which were supposed was supposed
to be able to not change the code
that much.
In practice, distance
is not possible for
every small detail, but for large
parts of the code.
So there are these two approaches.
And just to tell you, if you're
interested about it, here is
a list of literature.
And I think this paper
describes it in detail.
Something to drink.
I have a couple of questions still.
We have some time, number eight, please,
here in the back. Sorry we can't see you
really. And thank you for the talk.
And you stated that most
of your calculations were for climate
models. Do you think it's feasible
to apply these to operational
short time weather
models?
OK, so to
develop that Deepu version
was actually a lot of it was driven
by Matthew Suess, which is that the Swiss
Weather Service and
a lot of the developers were paid by
Metro Swiss and.
The initiative, I think a bit of it was
also born there because
today they run
this code that I showed you, the
broader away from, they run that
operationally.
They do, I think
around 32 kilometer simulations and
one one kilometer, say,
simulation operationally,
daily.
And currently the code is reintegrated
into the master branch.
And I'm I know that several
other weather services
want to pick it up because for them
it's much less about
large remains. But for them it's about
the energy use.
So you you you
by switching your architecture, you
save a lot of energy.
And if you have to pay the energy bill
yourself, the investment
into the engineers will be compensated
quite fast.
A number four in the back.
Thank you. Um, I have four questions.
Um, first
before. Yes, firstly
one.
Yes, first.
Um, the time to resolution seems
to be very high.
What's the practical limit
in that?
And secondly, what's the
mean time between failures for
a machine like the, uh, like, like
the ones you run into?
The third question, what's
the minimum resolution that you expect
to be useful?
Will it be a qualitative plateau
eventually and forcedly?
And this is a little different.
Um, well,
how about the competition between
models and the parameterization?
How much does politics play a role in
that? I mean, in the sense of
international politics and the
competition between different
climate models coming from different
nation states.
OK, I will answer the first three to
the last one and will take a long time.
We can talk about that in between.
So, yes, I'm just three minutes.
Yeah. Minimum time to solution.
So it is the latency of
an architecture so
that the latency of the memory access,
so that will
there will be some physical limits
on how fast you can access
a memory.
So that's why we think it will be very
hard to go to to ten
or twenty simulation years.
But they because today we apparently
can't see an architecture that the
latency is short enough.
Um, the second question,
what was it? Can you help me again?
Are you still there?
Yes. Yes.
What's the meantime.
Twin failures for supercomputer.
Um, I don't know the
number from the top of my head, from
my experience from doing these
simulations is I would
say.
Each month, so we did this in monthly
chunks and there was say
it was about 10 percent
and it was mostly
this access that failed
mental. Suess has two computers,
so whenever one fails, they switch over
the entire thing to another one.
So they have backup systems, I
don't know, in percent, but I think
they're at least are very robust.
And the third question was, again,
dirt. Sorry I'm so nervous.
I can't remember the state of plateauing
resolution. Oh, yeah.
And so the ultimate resolution that we
want to go through two is ten
to the minus two meters.
So zero point one centimeter because
that's the that's the
that's kind of the
inertial sub range of turbulence.
I see nice, nice crowds arriving
here in the meanwhile, I have a question
number five, I'm sorry, cruel
intervention. Thank you for your
thank you for the talk.
So you're
welcome
having it all.
Thank you for the talk.
So my my question is on
the on the grid side, so with
smaller grid size, you have
to decrease the time stepping
as well.
So and you using
a weather model, as far as I understand,
to do climate modeling.
So we know your problem is that the
timestep is getting smaller and
your, you know, numerical errors add
up.
And in fact, if you do that over
30 years, your your
results will be some kind of of
full of numerical errors.
So why not do a very good
simulation for one year
to a parameterization
and use the parameterization
for the climate models again and
to adapt the grid,
blah, blah? Yeah,
that's a nice question.
Thank you. OK, do we have
enough time?
Can I try to answer that.
OK in one minute.
So we will solve it all.
So in numerical schemes you have explicit
schemes on your explicit schemes and
in the explicit schemes.
Yes. You have to decrease that your time
step with the
with the grid spacing.
But the cool thing is that the methods
are only local.
So they're the perfect linear scaling
that I showed you is because the
methods only involve local
communication and many implicit
methods have global communication.
So if you think of it in terms of
the energy question, again, the energy
consumption is a lot higher
then coming to the numerical errors.
Of course, the numerical error is
given by the grid spacing
and a bit less by
the small time steps, because
first of all, we have a lot
of forcing.
We have the sun, which is a very
strong forcing. We have continents that
force. We have mountains
that impose forcing.
We have very large dynamical
systems that impose forcing.
So d the first thing
when you compare the strong forcing with
the air or the first single dominate,
then the other thing is that we have
something called a chaotic system.
So the Loreen's papers, if you if you
know what I'm talking about.
So very small errors
can cause very large
differences in the simulation after
a couple of days.
And we have by now we
have some methods how to address this
chaos problem, to which the
solution is we do a lot
of simulation and perturb them
a bit.
So in the end, they will lead to will
have a climate which is in
the mean, very similar.
But the initial
state of the state at
a certain time step is very different
from each other. So this is a lot of work
by Tim Palmer and his group.
You're interested? I can recommend some
papers that well,
thank you very much for this fantastic
lecture.
Oh.
We're glad to have you there to take
a look at. This is Jamie.