Identifying optimal catalyst materials for specific reactions is essential for advancing energy storage technologies and promoting sustainable chemical processes. To effectively screen catalysts, scientists need a deep understanding of systems’ adsorption energy—a vital area where machine learning (ML) models, especially graph neural networks (GNNs), have excelled in predictive accuracy.
However, GNNs are characterized by high inductive bias and necessitate atomic structures for their analysis. While this technique can effectively encode the atomic structure, it falls short in leveraging experimentally observable features.
Building on groundbreaking research that demonstrated a transformer-based language model’s proficiency in predicting adsorption energy using only human-readable text—without requiring complex preprocessing—researchers at Carnegie Mellon University‘s Department of Mechanical Engineering have developed a novel methodology that enriches this model through multimodal learning.
“When it comes to chemical data, by integrating different modalities, we can construct a comprehensive view. Inspired by this, we used multimodal learning to improve the performance of the predictive language model,” explained Janghoon Ock, a Ph.D. candidate in Amir Barati Farimani’s lab.
Their approach facilitates essential connections between various model setups, significantly boosting the language models’ capacity to accomplish prediction tasks without the need for specific labels. This self-supervised process, known as graph-assisted pretraining, achieves an impressive reduction in the mean absolute error for energy predictions regarding adsorption configurations by 7.4-9.8%.
Featured in Nature Machine Intelligence, this methodology skillfully integrates a generative language model into the framework, eliminating the dependency on traditional structural information.
The training process consists of two steps: graph-assisted pretraining and energy prediction fine-tuning.
The training process consists of two steps: graph-assisted pretraining and energy prediction fine-tuning. Credit: Carnegie Mellon University
As a result, the researchers enable the system to estimate initial energy predictions without depending on atomic coordinates.
“My ultimate goal is to build accessible and interactive methodologies that non-computational scientists can use,” said Ock. “LLM can be a key to achieving that accessibility and interactiveness. While it’s not the case right now, we are moving in the right direction.”
“Being able to produce energy estimates with just chemical symbols and surface orientations is a leap forward for accessible ML models,” emphasized Barati Farimani, associate professor of mechanical engineering.
The team plans to create a more extensive language-oriented platform for catalyst design in the future by adding more functional tools and enhancing the platform with reasoning and planning abilities in a framework similar to that of an agent.
Journal reference:
Janghoon Ock, Srivathsan Badrinarayanan, Rishikesh Magar, Akshay Antony & Amir Barati Farimani. Multimodal language and graph learning of adsorption configuration in catalysis. Nature Machine Intelligence, 2024; DOI: 10.1038/s42256-024-00930-7