Radu Popescu

Improving the Write Scalability of the CernVM File System with Erlang/OTP

CVMFS Developer, Fellow @ CERN

Improving the Write Scalability of the CernVM File System with Erlang/OTP

The CernVM File System (CernVM-FS) provides a scalable and reliable software distribution service. It was developed at CERN to assist High Energy Physics (HEP) collaborations in deploying software on the worldwide distributed computing infrastructure for data processing applications, but it can also be used in other domains.

CernVM-FS is implemented as a POSIX read-only file system in user space (FUSE) and uses content-addressed storage and Merkle trees for maintaining file data and meta-data. Files are stored remotely on standard web servers and are retrieved and cached on-demand through outgoing HTTP connections only, avoiding most of the firewall issues of other network file systems.

For writing, CernVM-FS follows a publish-subscribe pattern with a single source of new content that is propagated to a large number of readers.

This talk focuses on a new publication architecture, capable of accommodating dynamic strategies for content publishing and improving the scalability of the publication phase. We chose Erlang/OTP for implementing certain components of the new architecture, due to its suitability for making scalable and fault-tolerant systems.

Talk objectives:

  • Give an overview of the CernVM-FS project.
  • Show a case study of using Erlang/OTP as part of a network file system architecture.

Target audience:

  • People who are interested in Erlang, distributed file systems and systems programming.
  • People who need to distribute many, relatively small, files to many places.

About Radu

Radu Popescu is currently a CERN fellow, working on the CernVM-FS distributed file system project, a mission critical system of the CERN physics collaboration. He really enjoys distributed systems, expressive type systems and functional programming. He believes studying programming language design makes for a better programmer, regardless of the tools used. Radu has worked on a multitude of other interesting projects, including a 3D editor for photogrammetry data - Pix4Dmapper and libraries for high performance computing (HPC) - Trilinos, LifeV. His relationship with HPC culminated in a PhD degree obtained at the Ecole Polytechnique Federale Lausanne, Switzerland.

Github: radupopescu

Twitter: @iradupopescu

Back to conference page