KNMI projects ECMWF MSCR-31 Highlights

On this page

MSCR-31

ECMWF

ECMWF

Highlights from MSCR, October 2019

Artists' impression of the new Bologna data centre.
Image: gmp von Gerkan, Marg & Partner
This year's meeting of the ECMWF Member State Computing Representatives (MSCR-31) took place from October 23 – 25 at ECMWF in Reading. The meeting was attended by 21 Member States and Co-operating States, representatives from CTBTO and EUMETSAT, and some 24 people from ECMWF itself.

On this page you will find some highlights which might be relevant or otherwise interesting for KNMI and other Dutch users of ECMWF computing facilities. More details can be found in the presentations, which are available on the Internet (see the menu). There you can e.g. find some statistics, an account on the developments of various web services and examples of the possibilities of the new dissemination editor.

BOND (Bologna Our New Datacentre)

Work on ECMWF's new data centre in Bologna is well on its way, although unforseen ground conditions have caused delays. A total of 15 ITTs are needed to procure all necessary equipment.
By September 2020 the Centre will have 25 staff working in Bologna.

For users of the ECMWF facilities, the move to the new data centre will not go unnoticed:

HPC and ecgate services

The HPC and ecgate services will be merged into one new HPC. That means that everyone will have access to only one machine in future — with one batch system and one software stack. There will be some separation between scalar and parallel work, but how that is going to look and what they will do with accounting is still open.
A general user migration is expected 3 – 4 months before the operational switch-over, so towards the end of 2020.

MARS and ECFS

Hardware and tape media for MARS and ECFS will be physically moved to Bologna ahead of the new HPC. There will be extra disk space available in Reading during the move to cache data that is frequently used, but that can obviously not hold everything. Data on tape will be unavailable for weeks or even a few months until all tapes have been re-loaded in the new data centre. Moreover, it might be expected that some tapes get damaged during transport and that some data will be lost.
For 2 – 4 days there will be no MARS or ECFS service at all and the bandwidth for data retrievals will be reduced for several months.

Actions requested from us

Software

Not all of the (older) software is going to be available on the new servers and the plan is to maintain just a few recent versions of all of the software packages there. Check ECMWF Confluence for a list of software that is going to be phased out.

Copernicus Climate Change Service

ECMWF hosts the Copernicus Climate Change Service (C3S). C3S supports society by providing authoritative information about the past, present and future climate in Europe and the rest of the World.

Climate Data Store (CDS) and Application Data Store (ADS)

CDS is a data store that brokers access to datasets maintained by third parties. It offers data retieval, processing and visualisation through a web platform and CDS API.
Datasets are assigned a DOI to offer an easy way to reference.

ERA5

ERA5 was one of the first data sets to become available in CDS. For public users CDS is now the only access point for ERA5 data. Web API access has been closed. Member state users can still access ERA5 data through MARS as well.
Surface and pressure level data are stored on disk in CDS, but model level data have to be retrieved.

European Weather Cloud

ECMWF and EUMETSAT are developing the European Weather Cloud (confusingly for KNMI: EWC) to enhance the use of their data, increase the socio-economic benefits, and exploit opportunities and address challenges linked to open policies and market changes. The value lies in fast access to data, which seems to be only possible realistically and sustainable with cloud technology.
In Januari 2019 a pilot project started to build a private cloud, and gain experience with use cases. The coming year they will continue with the implementation. New use cases are still welcomed, they will help in define what is needed on the European Weather Cloud.

Software developments

Python 3

ECMWF software (ecFlow, ecCodes, Magics, Metview) is now available for Python 3. The aim is to have ECMWF software available in all mainstream package managers.

ecCharts

The interactive web application for forecasters to access the latest ECMWF forecasts, ecCharts, has undergone a major upgrade. The new version, which became default in November, is significantly faster than before.

Github and code contributions

ECMWF has moved its source code to GitHub now. Apart from all the usual advantages, this makes it also easier for others to contribute.
So, if you have anything to contribute, code, documentation, tests, examples, etc, please do not hestitate to. See the link in the menu on the right.

The future of Numerical Weather Prediction

To improve forecast skill as before, the solutions from the past do not work anymore. We can not pack more transistors on a chip, and clock-speed does not increase anymore. Moore's law and Dennard scaling have come to an end. Furthermore, NWP codes have become increasingly inefficient on HPC systems, because of increasing forecasting system and code complexity. Bigger computers, the solution of the past, will be simply unaffordable to buy and unaffordable to power. Handing the computer codes to software people to make them faster will not work because the codes lack in flexibility and significant scientific judgement will still be needed. Also data sizes and hence IO is growing exponentially.
To meet these challenges, ECMWF set up the Scalability Programme several years back. With new technologies for processors, IO and integration they now feel well on their way to move forward.
Hans de Vries, © KNMI, December 16, 2019