Dev Tool for Data Scientists: Using Supercomputers - Made Easy

“You need people who have the courage to head towards uncharted territories.”

Why we like showing this case

It feels good to make life a little easier for yourself and build a tool that makes you work faster and more efficiently.

It feels even better when you can also make life easier for others and have the chance to expand your own tool into a marketable Data Science devtool. Here we could perfectly combine our two competences - software development and data science - because the developers and the users of Mantik are sitting in the same room at our company anyway!

Task and solution

In 2016, our Data Scientists identified a number of steps that kept coming up in Machine Learning projects, but that no one felt like doing because they were repetitive and time-consuming. Big players like AWS and Microsoft Azure started serving the needs of machine learners shortly after, but many of them still remained unanswered:

Results were not sufficiently reproducible, because to do so, you need to take a holistic view of the ML application: A model consists of the triple of data,code and hardware. They belong together and determine the applicability of ML algorithms. Collaborative work on the same model, even more so across different environments, was hardly supported. There was no way to quickly or even automatically test and benchmark models against each other. It was costly to transfer models developed in Python into languages that performed better in production environments, and deploying trained models could be a real pain.

Ambrosys began designing a tool that could serve as a backbone and daily helper for millions of Data Scientists. Unlike the proprietary tools of the US software giants, Mantik is open source to activate the open source development power of the Data Science community.

In 2020, we took stock. The architecture was in place, as were interfaces to key ML libraries, and reproducible training runs were possible. But the rest of the world continued to spin as well; many new tools were entering the market. We couldn't fall into the trap of trying to pack too much functionality into our product and become too slow as a result. We developed an architecture that allowed us to integrate new components instead of reinventing things. For example, we combined Mantik with the full-featured ML Flow (a wheel we didn't have to reinvent) and consistently unfolded it towards a worthwhile niche: High Performance Computing.

Mantik's mission is now to provide large and largest computing clusters like JSC in Jülich or CSCS in Lugano as effortlessly as an AWS cloud. Currently, two large-scale EU research projects (MAELSTROM and KI:STE) are finalizing Mantik v2.0.

→ More about Mantik

About High Performance Computing

High Performance Computing, HPC or sometimes simply called "supercomputing", offers tremendous opportunities where extremely large amounts of data need to be processed in a manageable time frame. Typical applications are those dealing with particularly complex systems, such as weather and climate research, aerospace or financial mathematics. Machine learning and HPC are also actually ideal partners; because HPC-ML is a young discipline and few convenient tools are available, far fewer users access HP resources than is actually possible or useful.

Ambrosys is one of the pioneers of those opening up HPC to new application areas. We are excellently networked with JSC Jülich and other high-performance computing centers and can support both the development of HPC applications and the booking and use of external HPC capacities.

→ Questions or thoughts about High Performance Computing? Dr. Markus Abel is looking forward to hearing from you.