As part of the efforts to continuously enhance the user experience, a number of new functions and improvements have been added to the Destination Earth Data Lake. These features facilitate data discovery and resource and workflow management, while also extending knowledge sharing through AI tooling.

Implemented by EUMETSATEuropean Organisation for the Exploitation of Meteorological, the Data Lake is one of the central components of Destination Earth (DestinE). It provides access to enormous volumes of data from the DestinE Digital Twins and numerous other data spaces, with its distributed infrastructure allowing users to process that data near to where it is produced. Through a series of development increments, the DestinE DataEarth-system monitoring and prediction system output as prov More Lake continues to evolve in order to afford users more services, tools and applications to address their Earth-system queries.

In the latest increment, a significant set of new key features has been added to the Data Lake, creating tangible benefits for users. Some of the highlights of these releases are outlined below.

Automated HDA API connections

The Harmonised Data Access (HDA)Logical function supporting a standardised API that provides More service enables Data Lake users to request DestinE and other data. Its ApplicationAn action, information product, or service (including consul More Programming Interface (API) provides connections to both users’ programme workflows and applications, and new API keys allow applications and HDA to communicate securely to automatically retrieve and process data.

Improved data collection discovery and broader data offering

The Data Portfolio within HDA is a catalogue-based interface for discovering data collections through high-level metadata, such as thematic focus and temporal coverage. In the latest release, improved filtering of data collections gives rise to faster, more accurate, more inclusive data discovery. Enhanced filters reduce search effort, support a wider range of user profiles, improve comparability between datasets and ensure the Data Portfolio remains usable and scalable as it continues to grow.
External data providers’ products can now be fetched through the Data Portfolio. The additional 44 data collections bring the total to 241.

New AI widget interactive assistant

The Data Lake now includes an early-release interactive assistant or chatbot, designed to help users navigate more easily and understand what data and services are available. Users can ask questions in natural language, and the assistant gives immediate guidance on data discovery, service capabilities and practical examples. This reduces the need to search through extensive documentation and multiple information sources, helping users reach relevant information more efficiently.

Advanced News section

The News section on the Data Lake website is a new dedicated space for clear and timely sharing of updates on data, services and features. This helps the user community to stay informed about developments and better understand how new capabilities can support their work.

Improved resource management

Data Lake Edge Services are accessed through approved projects. Submitting projects for approval, along with the subsequent management of resources, users’ permissions and Data Lake sites, is carried out with the My Data Lake Services tool. With the latest release, users can now work on several projects with the same account, and each project is able to request Data Lake resources at multiple locations. This simplifies coordination and resource management across the Data Lake’s European infrastructure. Additional improvements include easier switching between projects and clearer governance of project membership.

GPU-enabled environments

Graphical Processing Units (GPUs) are essential processors for working with machine learning models. In the latest release, the DestinE DataEarth-system monitoring and prediction system output as prov More Lake now has GPU-enabled JupyterHub environments, which allow users to run computationally intensive analyses and machine learning workflows much faster. Such high-performance processing is critical when working with large Earth observation and Digital Twin datasets.

With the release of these new Data Lake features and tools, users are further empowered to accelerate experimentation and the generation of insight on our planet’s systems. An additional release will take place this summer, and the continuation of DestinE into its Phase Three development, as announced recently, will bring ever more improvements to the Data Lake as it continues to mature.

More details on the latest feature release can be found on the Data Lake website.