From CERN to Cledar
The story of two engineers, their experiences at CERN, and the birth of Cledar.
CERN is one of the world’s largest and most-respected centers for scientific research and home to the world’s largest machine – the Large Hadron Collider (LHC). Physicists at CERN generate many petabytes of data smashing particles to reveal the origins of the universe.
CERN is also the birthplace of something called the Internet and the start of an idea that would grow to become Cledar.
Cledar’s founders – Hubert and Piotr – spent many of their formative years at CERN and contributed to some of the greatest scientific discoveries of the 21st century.
The Globe of Science and Innovation at CERN, the European Center for Nuclear Research.
Chief Technology Officer
Chief Executive Officer
Combined, Piotr and Hubert spent more than fifteen years at CERN. During this time, they applied their skills to data science, data analysis, software development, and infrastructure operations – all in the name of scientific progress and finding meaningful insights in petabytes of data.
It was at CERN that Piotr and Hubert met, created the vision for Cledar, and honed the skills and expertise that, today, are helping organizations of all shapes and sizes channel and maximize the value of the data that surrounds them.
In 2002, Piotr was nominated to join the prestigious Academic Computing Center Cyfronet (Part of the AGH University of Science and Technology) to work on the newly started Computing Grid “CrossGrid”, which was part of a larger collaboration linked to the Worldwide LHC Computing Grid (WLCG) and CERN. Piotr set up the very first computing cluster in Central and Eastern Europe connected to the worldwide computing grid.
Piotr in front of the Atlas LHC detector in Meyrin, Switzerland.
Hubert stands before the data room in building 190 at CERN in Meyrin. The first data from beam tests of the prototype TOTEM LHC detectors was consolidated and analyzed there.
At the request of a family friend and professor at Warsaw University of Technology, Hubert traveled to CERN for the summer to help with data analysis on the TOTEM experiment at the Large Hadron Collider (LHC). Hubert prototyped detectors to handle the data generated by 40 million collisions per second. To ensure the successful progression of the TOTEM project, it was Hubert’s job to find meaning in the data being generated.
After the successful TOTEM project, Hubert was again recruited by CERN to “simulate the collider”. The physics team needed to be sure that the data generated within the accelerator would be captured accurately, but without quite knowing what would happen. Hubert was essentially tasked with anticipating physics that had never been measured before.
Totem Roman Pot detectors in the LHC. Hubert simulated the LHC running conditions to make optimal use of the hardware infrastructure and carried out analysis of the data captured.
Nodes of the very first computing grid cluster, made from desktop computers, at CERN’s Meyrin data center.
After just two years at Cyfronet, Piotr was recruited to CERN, and he began work there in April 2004. Being responsible for grid monitoring and operations, Piotr was intrigued to observe that, while many teams had their own grid monitoring tools reporting healthy statuses, complaints were mounting about jobs not running or completing properly.
Piotr developed an agent system to automate tests of the vast grid and report back with findings. The implementation of this system led to the disturbing revelation that only about 20% of the grid was, in fact, healthy (green), while the rest was coming back with various errors.
While officially known as “Service Availability Monitoring” (SAM), CERN personnel colloquially referred to the SAM’s as “Piotr’s tests.” This achievement was like a badge of honor, but it helped to foster the Cledar culture of questioning the fundamentals at every opportunity.
Hubert and Piotr met in 2004, over a meal with mutual friends, where Hubert learned that Piotr was to blame for his computer processing jobs not coming back complete. They immediately hit it off, and it was not long before ideas for future collaboration began to emerge.
Restaurant R1 in Meyrin, where Piotr and Hubert had many heated debates about technology.
Hubert and Piotr in Sergy, France, close to Hubert's house.
The work and aggressive deadlines were occasionally interrupted by informal gatherings at cafés, late-night parties in attics, and mountain hikes in the gorgeous countryside on either side of the France-Switzerland border. The surroundings and the culture created by the coming together of some of the world’s brightest minds served as the perfect breeding ground for the ideas and inspiration that would shape future events.
One of the key sources of inspiration were the regular adventures into the Swiss mountains. There, in the municipality of Meyrin, Piotr and Hubert learned the local word for a special type of gate that leads into the mountains. Cledar.
For 4 years, Hubert’s team used every data algorithm, optimization technique, and ML/neural network at their disposal to simulate different scenarios (e.g., different beam conditions, various detector setups) and arrive at a full-specification detector setup (detector distances to the beam, performance of silicon wafers, shapes of magnetic fields, etc.). The team also contributed significantly to designing LHC running conditions.
The work led by Hubert meant that the team of scientists avoided most of the pitfalls and challenges that typically precede the capture of quality data. Instead, they benefited from high-quality results within a week of operations starting, thereby accelerating their journey to achieving their goals.
Prototype of the Roman Pot detector before installation in the LHC. The detector was designed to withstand the energetic LHC beams at a distance of a fraction of a millimeter.
Data originating from all LHC collision points was collected at the CERN data center, consolidated, and made available for further analysis in the Worldwide LHC Computing Grid.
Piotr’s team was responsible for ensuring that every team at CERN had access to the tools it needed, that these tools and teams were connected to the vast technology infrastructure at CERN, and that data was captured and processed in the right way. This enabling role was played in a time before virtualization and clustering were mainstream solutions, and before cloud software deployment techniques like containerization. Piotr’s team thrived in developing solutions that enabled scientists to focus on achieving their goals, without being limited by technology.
There were two interlinked projects. The first, the Worldwide LHC Computing Grid (WLCG), focused on providing computational and data storage infrastructure for LHC experiments (Atlas, Alice, CMS, LHCb), spanning world-wide data centers. The other project was Enabling Grids for E-science in Europe (EGEE), which provided multidisciplinary scientific infrastructure (not only LHC but also in medicine, climate sciences, biochemistry, etc.).
Piotr decided to leave CERN early in 2008. By the time he left, he had overseen infrastructure projects that meant the grid was now sufficiently scalable to support the growth of CERN-based research for years to come.
What had been known as “Piotr’s tests” officially became Service Availability Monitoring tests or SAMs and were integrated into a resource-brokering system that improved the organization and its ecosystem’s capabilities. Individual experiments could now exclude non-compliant nodes or sites, and the system was further integrated with automated alarms, notifications, problem-resolution tracking systems, and more.
When Piotr started working at CERN in 2004, there were 30 data centers with a couple of thousand CPU cores. By the time he left in 2008, there were over 300 data centers with over 50k CPU cores. Today it is one of the largest hybrid infrastructures in the world with over 170 data centers and more than a million CPU cores.
Over the same period and thanks to the system management and operational procedures that Piotr put in place, system availability increased from around 25% in 2004 to around 95% in 2008, generating savings of around 60m CHF annually.
Close-up view of computing nodes of the Worldwide LHC Computing Grid installed in Meyrin.
The LHC incurred serious damage as a result of unreliable electrical connections.
After years of preparation, the LHC was launched amid strong global interest. However, on September 10, after just 10 days, an explosion caused by defective electrical connections put the LHC out of action for several months. The energy stored in the accelerator at the time of the explosion was comparable to that of a high-speed train.
Hubert, fresh from defending his PhD thesis, contemplated the event with classic pragmatism. “We have more time to refine our physics program”.
Hubert became Head of Analytics for the TOTEM physics team and coordinated the analytics of physics related to proton scattering, which revealed three main data challenges: 1) magnetic field uncertainty in the particle collision area, 2) erratic beam positioning, and 3) excess background noise.
Solving these challenges enabled the most precise (to date) measurement of the proton diffractive structure. Hubert’s data set also enabled the discovery of a new particle (hadron), called an Odderon.
Large Hadron Collider (LHC) installed in a tunnel measuring 27 kilometers beneath Switzerland and France.
Piotr with the Golden Gate Bridge in the background.
After several years of working in the European sphere of science and academia, Piotr began work for a leading global provider of virtualization solutions based in California’s Silicon Valley, where he served as Senior Software Engineer and Project Manager, and later, Head of the Krakow office.
This adventure afforded Piotr the opportunity to gain valuable experience in the private sector, refine his skills, enhance his understanding of contemporary business challenges, and expand his network. After leaving this role, Piotr also undertook various consulting and project-based commissions in the public and private sectors.
The seed of an idea that was Cledar, planted several years earlier in Switzerland, continued to grow, enriched by these new experiences.
After many successful projects and collaborations at CERN, Hubert felt it was time to apply his now-extensive expertise to the private sector and return to his native Poland.
Cledar, having been formed earlier by Piotr with several other former-CERN scientists, was ready for a full-time CEO who could lead new ML projects, build teams, and create a healthy structure for the growth of the organization.
Hubert and Piotr’s experiences at CERN helped shape Cledar as a company. Today, Piotr continues to work with challenging data infrastructures and Hubert continues to conceptualize solutions in data science. With this core expertise and the addition of other experts, Cledar continues to grow from strength to strength, although Piotr and Hubert look back at the years at CERN with fondness and are grateful for the opportunities their time with CERN afforded them.
Hubert and Piotr discussing challenges and ideas for the future.
You’ve learned about our past, now learn about Cledar today, what we stand for, and how you can become part of Cledar’s future.