当电子显微镜遇上AI：自动化实验引领科技新潮流

材料科学主要研究材料的结构与性质之间的关系，这些关系跨越从原子到微米尺度。扫描透射电子显微镜（STEM）已成为在这些尺度上研究材料的重要工具，特别是由于其能够与先进的数据分析技术相结合，为自动化实验和多维数据处理提供了新的机遇。

Fig. 1 Example of the errors introduced via StyleGAN116.

随着机器学习算法的发展，STEM在实时分析和自动控制方面的应用前景广阔。由美国田纳西大学材料科学与工程系的Sergei V. Kalinin教授和橡树岭国家实验室计算科学与工程部的Debangshu Mukherjee博士领导的团队，对扫描透射电子显微镜中的自动化实验机器学习进行了综述。

Fig. 2 Automated task-based statistical analysis via few-shot learning.

扫描透射电子显微镜及其光谱技术已经成为现代材料科学、凝聚态物理、化学和生物学等领域的基石工具。这项技术的影响力与其能够洞察材料结构和性质的量化信息量直接相关。无论是冷冻电子显微镜（Cryo EM）还是小晶体电子衍射等领域的突破都表明，数据分析方法和高效的操作流程极大地提高了从技术发展中所获得的价值，并显现出该领域的巨大增长潜力。

在STEM领域，自动化实验的发展是一个快速崛起的趋势。目前，正从人工实验向自动化实验的过渡期中，面临着众多挑战。在仪器方面，需要发展高级别超语言，以便用最基础的操作单元描述人类动作。在机器学习领域，这要求开发出对分布外漂移效应具有鲁棒性的监督学习算法，以及能够在少量数据上训练的主动学习技术。

在计算和网络领域，这需要构建边缘计算基础设施，不仅能够支持快速分析和决策，还能将仪器接入全球云网络。这一点将进一步推动高效的数据与代码共享，形成分布式的人机协作团队，并催生出跨仪器的网络协作平台。

尽管如此，向自动化实验的转变同样要求科学界在规划与实施实验活动方面作出根本性的改变。目前为止，已知的所有显微镜自动化实验均采用基于固定策略和预先定义的兴趣对象的工作流。仅有的超越传统人工操作流程的实验案例，是那些基于深度核学习的逆向发现实验。

要真正释放自动化实验的潜能，关键在于明确定义实验激励，即明确的实验目标，这可以是探索性发现、假设验证或定量测量等。许多这样的激励目标通常只在特定领域应用的更宽广的科学背景中才能被界定。接下来，需要制定确定性或概率性的策略，即将以超语言表达的具体行动与系统的当前状态（图像或光谱）连接的算法。这些策略可以在实验前设定，以协调探索和利用之间的目标，或者更引人注目的是，策略可以随着实验的进行而不断发展，以便在既定实验预算内实现既定的奖励目标。

综上所述，STEM领域的自动化实验（AE）虽处于起步阶段，但变化迅猛。鉴于基于Python的API和云基础设施、远程控制的显微镜的迅速发展，尤其是考虑到贝叶斯优化、强化学习以及其他随机优化形式等主动学习方法的最新进展，可以预见该领域将在未来几年内迎来快速增长。

该文近期发表于npj Computational Materials 9: 227 (2023).

Editorial Summary

Materials science focuses on the study of the relationships between the structure and properties of materials that span from the atomic to the micrometer scale. Scanning transmission electron microscopy (STEM) has become an important tool for studying materials at these scales, especially due to its ability to be combined with advanced data analysis techniques, which provide new opportunities for automated experiments and multidimensional data processing. With the development of machine learning algorithms, STEM has promising applications in real-time analysis and automation. 

A team lead by Prof. Sergei V. Kalinin from Department of Materials Science and Engineering, University of Tennessee and Dr. Debangshu Mukherjee from Computational Sciences and Engineering Division, Oak Ridge National Laboratory, USA, reviewed machine learning for automated experimentation in scanning transmission electron microscopy. Scanning transmission electron microscopy and spectroscopy has become one of the foundational tools in modern materials science, condensed matter physics, chemistry, and biology. The impact of this technique is directly related to the amounts of quantifiable information on materials structure and properties it can derive. The success of fields such as Cryo EM and small crystal electron crystallography suggest that the availability of the data analysis methods and operational workflows greatly amplifies the value derived from technique developments and suggests tremendous potential for the field growth. One of the rapidly emerging trends in STEM is the development of the automated experiments. 

Here, the authors overview some of the challenges that transition from human-driven to automated experiment EM will bring. On the instrument side, this necessitates the development of the instrument-level hyper-languages that allow to represent the human operations via minimal primitives. On the ML side, it requires development of the supervised ML algorithms that are stable with respect to the out of distribution drift effects and active learning methods that can be trained on small volumes of data. On the computational and network side, it requires development of edge computing infrastructure capable of supporting rapid analysis and decision making, and connect the instrument to the global cloud. The latter in tern opens the pathway to the effective data and code sharing, formation of the distributed human-ML teams, and emergence of the lateral instrumental networks. However, the transition to the automated experiments also requires deep changes in the way scientific community plans and executes experimental activities. To date, all examples of the automated experiment in microscopy the authors are aware of are performed with the workflows based on fixed policies and a priori known objects of interest. The only examples of beyond human workflows include the inverse discovery experiments based on the deep kernel learning. Going beyond simple imitation of human operation and unleashing the power of automated experiment requires clearly defining the experimental reward, i.e. specific goals. This can include the discovery (curiosity learning), hypothesis falsification, or quantitative measurements. Many of these rewards are defined only within a broader scientific context of specific domain applications. Secondly, this requires formulating the deterministic or probabilistic policies, i.e. algorithms connecting the specific action expressed in the hyper language and the observed state of the system (image or spectra). These policies can be defined prior to the experiment to balance the exploration and exploitation goals. Alternatively, and much more interestingly, the policies can evolve along the experiment to achieve the desired reward within the given experimental budget. 

Overall, the current state of the AE in STEM is nascent but fast changing. However, given the rapid emergence of the Python-based APIs and cloud infrastructure, remotely controlled microscopes, and especially given recent advances in active learning methods including Bayesian Optimization, reinforcement learning, and other forms of stochastic optimization, this field is likely to grow quickly in the coming years.

This review article was recently published in npj Computational Materials 9: 227 (2023).

原文Abstract及其翻译

Machine learning for automated experimentation in scanning transmission electron microscopy（机器学习在扫描透射电子显微镜自动实验中的应用）

Sergei V. Kalinin, Debangshu Mukherjee, Kevin Roccapriore, Benjamin J. Blaiszik, Ayana Ghosh, Maxim A. Ziatdinov, Anees Al-Najjar, Christina Doty, Sarah Akers, Nageswara S. Rao, Joshua C. Agar & Steven R. Spurgeon 

Abstract Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centric experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows, as well as the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation.

摘要 机器学习（ML）已成为（扫描）透射电子显微镜(S)TEM成像和光谱数据后期处理的关键技术。目前的一个新趋势是向实时分析和闭环显微镜操作的过渡。在电子显微镜中有效利用机器学习现在需要开发以显微镜为中心的实验工作流程设计和优化策略。在这里，我们讨论了向主动机器学习过渡所面临的挑战，包括顺序数据分析、分布外漂移效应、边缘运算要求、本地和云数据存储，以及环路理论操作。特别是，我们讨论了人类科学家和机器学习代理在实验工作流程的构思、协调和执行中的相对贡献，以及开发可跨多个平台应用的通用超级语言的必要性。这些考虑将共同影响机器学习在下一代实验中的操作化。