Executable books make campaign data tangible

Presentation slides: https://eurec4a.pages.gwdg.de/dach2022_howto_eurec4a/

Theresa Mieslinger, Jule Radtke and Tobias Kölling

Max Planck Institut für Meteorologie & Universität Hamburg

Wealth of data …

… collected in large field campaigns like EUREC⁴A (Stevens et al. 2021).

• Field campaigns are dedicated to study our earth system, thoroughly quantifying the state of a specific region in space and time on our planet.
• We had the honour to participate in the eurec4a field campaign (to study trade cumulus, North Atlantic Trades around Barbados), great number of research platforms
• These create a great wealth of original datasets

Wealth of data …

… and distributed among many research institutions.

• only made possible as a number of groups of people from many research institutions participated (e.g. aircrafts)
• thus scattered around the globe, distributed data
• data, especially the wealth of data together, is potentially very useful for model validation, process studies, teaching and more.
• But while researchers way beyond those participating at the field campaign might be interested in the data, even those participating are often faced by difficulties to discover, access and use this distributed data.

How to … make data usable?

• Our question, motivation: How to make this wealth of data usable?
• to make data usable, questions like … need to be answered: data user, data provider side
• We exploit the recognition that by creating analysis scripts, that are openly accessible and executable by everyone, a solution to all these questions needs to be and is found

How to EUREC4A

Shareable analysis scripts in form of an executable online book …

https://howto.eurec4a.eu

… and in doing so, it becomes the hub of a larger ecosystem.

• How to EUREC4A is:
• An openly accessible online book,
• written collaboratively
• explanations about the available instruments, data and typical usage patterns
• organised in chapters, notebooks
• collection of data quicklooks
• and an interactive plotting software at the same time

The technical ecosystem

• pythonic approach
• dashed part is visible to the user
• jb, intake, xarray
• xarray: python package providing a simple and efficiant way of working with labelled multi-dimensional arrays (NumPy-like). Simplyfies analysis and visualization.
• data files (bottom of graph) distributed globally
• protocol (S3, HTTP, …) to access them and another inferface “fsspec” that makes the below layer look like a file system and defines standard operations (ls, cd, mkdir)

Jupyter notebook - an online executable book

• book-like structure
• Markdown for narrative content, e.g. instrument details
• MyST for executable content, e.g. code examples

https://jupyterbook.org

• book strucutre via a _toc.yaml file
• text articles are written in standard markdown files
• executable notebooks are written in MyST to not have lengthy cell executiona and output commited on git which would make it messy and unreadable
• Markedly Structured Text (MyST): variant of md and optimized to contain executable code cells

Taking the pain out of data access and distribution

• cataloging system: for listing datasets
• drivers: opening instructions

https://intake.readthedocs.io/

• “Intake is a lightweight package for finding, investigating, loading and disseminating data.”
• contains two main components: a cataloging system and data loaders (drivers)
• def driver: translates yml entry into opening instructions (-> to_dask())
• we use intake-xarray. other Plugins such as intake-esm might be interesting to modeling community

The EUREC4A intake catalog

https://github.com/eurec4a/eurec4a-intake

• hierarchical, searchable structure of nested objects, in our case YAML files

EUREC4A intake catalog usage

import eurec4a
cat = eurec4a.get_intake_catalog()
ds = cat.dropsondes.JOANNE.level3.to_dask()
ds.plot.scatter("flight_lon", "flight_lat");

How intake eases data access * one node opens your way to all datasets * user side: no hard-coded data paths in scripts shared by your collegue * dataset creator side: updates are possible without annoyed users

Accessing remote data: OPeNDAP

OPeNDAP provides software which makes local data accessible to remote locations regardless of local storage format.

• advantage: powerful server, allows access of data chunks
• disadvantage: error-prone

https://www.opendap.org/

• “OPeNDAP provides software which makes local data accessible to remote locations regardless of local storage format.”
• protocol based on HTTP and allows to access also parts of datasets Disadvantages: because the server is quite powerful (e.g. can convert between file formats) it is also error-prone. In our case, the downtime of the opendap-server exceeded by far the uptime -> not usable! Another disadvantage is the potential change of files on the server without you noticing. -> scripts will crash

Acessing remote data: the Zarr library

Zarr is a format for the storage of chunked, compressed, N-dimensional arrays.

• data in form of blocks or chunks
• single files allow for parallel processing

https://zarr.readthedocs.io

• can read and write arrays from multiple threads or processes
• simple, transparent, open, supports high-throughput distributed I/O on different storage systems

Acessing remote data: IPFS - the InterPlanetary File System

IPFS is a distributed system for storing and accessing files, websites, applications, and data.

• datasets can be pinned on several distributed servers
• IPFS searches for your requested dataset and delivers it from any of the close and running servers
• data is identified by it’s content (CID)

• decentralized file system
• rather object-based, not file-based (for Zarr, an object store is sufficient)
• CID: has the advantage that files cannot change without you noticing

https://ipfs.io/

Putting the puzzle back together

(our construction kit)

The social ecosystem

• the book does not only create and live through a technical ecosystem
• but also due to social feedback loops that are stimulated
• in the notebooks/scripts people use the data > test it > gets improved (i.e. making them even more analysis friendly) > …
• this cycle leads to a collaboration between data user and provider
• also book lives from and enables, fosters the collaboration of people > learning from each other > getting more engaged > working and learning from each other
• these social loops are one corner stone that make the book improve the usability of data

The social ecosystem

Thanks to all contributors!

Therefore: Thanks to all contributors

More examples…

https://howto.eurec4a.eu

• show toolbox flightsegmentation
• show specMACS example

Highlights

• goes far beyond FAIR data: fosters analysis-ready cloud-optimized data (Abernathy et al., 2021)
• data is used efficiently - duplicates are avoided
• inclusion
• collaboration and knowledge transfer
• scalable to meet future data needs

• Highlights of “how to make data usable”
• beyond FAIR: Findable, Accessible, Interoperable and Reusable data
• In doing so individual downloads creating inefficient duplicates that are not available to the community are avoided > data not locked in a fortress
• How to EUREC4A enables fast and easy access for everyone independent of access to computing centers and computing resources, thus promoting inclusion
• at the heart: share analysis scripts without moving tons of data and adjusting paths etc in script > fostering collaboration, knowledge transfer > social feedback loops

Conclusions

• an openly accessible and executable online book makes campaign data tangible
• explanations about the available instruments, data and typical usage patterns help to get started with the data
• as hub of a larger ecosystem, data becomes visible, easily accessible, shareable and usable
• the book lives from, enables and stimulates collaboration and inclusion
• positive feedback loops stimulate the publication of accessible and understandable datasets in an analysis friendly way