and prospective/current/enrolled master students in the fields of modern mathematics and computational science:
Scientific Learning
The Institute of Computational Science (ICS) at the Università della Svizzera italiana (USI) announces six seminars about Machine Learning, Neural Networks, Physics Informed Neural Networks
Topics
Machine Learning, Neural Networks, Physics Informed Neural Networks
PROGRAM
Time |
Speaker & Place |
May 19
5:00-7:00 pm
UTC+2:00 Zurich time
|
Prof. Paris Perdikaris
Lecture: 5:00-6:00 pm
Seminar: 6:00-7:00 pm
|
June 2
5:00-7:00 pm
UTC+2:00 Zurich time
|
Prof. Jonathan Siegel
Lecture: 5:00-6:00 pm
Seminar: 6:00-7:00 pm
|
June 16
5:00-7:00 pm
UTC+2:00 Zurich time
|
Dr. Eric Cyr
Lecture: 5:00-6:00 pm
Seminar: 6:00-7:00 pm
|
June 23
5:00-7:00 pm
UTC+2:00 Zurich time
|
Prof. Mishra Siddhartha
Lecture: 5:00-6:00 pm
Seminar: 6:00-7:00 pm
|
List of Speakers
Prof. Paris Perdikaris
assistant professor at Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania.
Part A:Lecture
Making neural networks physics-informed
Leveraging advances in automatic differentiation, physics-informed neural networks are introducing a new paradigm in tackling forward and inverse problems in computational mechanics. Under this emerging paradigm, unknown quantities of interest are typically parametrized by deep neural networks, and a multi-task learning problem is posed with the dual goal of fitting observational data and approximately satisfying a given physical law, mathematically expressed via systems of partial differential equations (PDEs). PINNs have demonstrated remarkable flexibility across diverse applications, but, despite some empirical success, a concrete mathematical understanding of the mechanisms that render such constrained neural network models effective is still lacking. In fact, more often than not, PINNs are notoriously hard to train, especially for forward problems exhibiting high-frequency or multi-scale behavior. In this talk we will discuss the basic principles of making neural networks physics informed with an emphasis on the caveats one should be aware of and how those can be addressed in practice.
Part B: Seminar
Learning the solution operator of parametric partial differential equations with physics-informed DeepONets
Deep operator networks (DeepONets) are receiving increased attention thanks to their demonstrated capability to approximate nonlinear operators between infinite-dimensional Banach spaces. However, despite their remarkable early promise, they typically require large training data-sets consisting of paired input-output observations which may be expensive to obtain, while their predictions may not be consistent with the underlying physical principles that generated the observed data. In this work, we propose a novel model class coined as physics-informed DeepONets, which introduces an effective regularization mechanism for biasing the outputs of DeepOnet models towards ensuring physical consistency. This is accomplished by leveraging automatic differentiation to impose the underlying physical laws via soft penalty constraints during model training. We demonstrate that this simple, yet remarkably effective extension can not only yield a significant improvement in the predictive accuracy of DeepOnets, but also greatly reduce the need for large training data-sets. To this end, a remarkable observation is that physics-informed DeepONets are capable of solving parametric partial differential equations (PDEs) without any paired input-output observations, except for a set of given initial or boundary conditions. We illustrate the effectiveness of the proposed framework through a series of comprehensive numerical studies across various types of PDEs. Strikingly, a trained physics informed DeepOnet model can predict the solution of O(1000) time-dependent PDEs in a fraction of a second -- up to three orders of magnitude faster compared to a conventional PDE solver.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Prof. Jonathan Siegel
assistant professor at the Department of Mathematics, Penn State.
Part A:Lecture
The Approximation Theory of Neural Networks
An important component in understanding the potential and limitations of neural networks is a study of how efficiently neural networks can approximate a given target function class. Classical results which we will cover include the universal approximation property of neural networks and the theory of Barron and Jones which gives quantitative approximation rates for shallow neural networks. We will also discuss approximation rates for deep ReLU networks and their relationship with finite elements methods. Finally, we will introduce the notions of metric entropy and n-widths which are fundamental in approximation theory and permit a comparison between neural networks and traditional methods of approximation.
Part B: Seminar
The Metric Entropy and n-widths of Shallow Neural Networks
The metric entropy and n-widths are fundamental quantities in approximation theory which control the fundamental limits of linear and non-linear approximation and statistical estimation for a given class of functions. In this talk, we will derive the spaces of functions which can be efficiently approximated by shallow neural networks for a wide variety of activation functions, and we will calculate the metric entropy and n-widths for these spaces. Consequences of these results include: the optimal approximation rates which can be attained for shallow neural networks, that shallow neural networks dramatically outperform linear methods of approximation, and indeed that shallow neural networks outperform all stable methods of approximation on these spaces. Finally, we will discuss insights into how neural networks break the curse of dimensionality.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr. Eric Cyr
Part A:Lecture
An Adaptive Basis Perspective to Improve Initialization and Accelerate Training of DNNs
Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We provide analysis of these techniques and illustrate via numerical examples dramatic increases in accuracy and convergence rate for benchmarks characterizing scientific applications where DNNs are currently used, including regression problems and physics-informed neural networks for the solution of partial differential equations. In addition, we present a partition of unity (POU) architecture capable of achieving hp like convergence. This methodology, introduces traditional polynomial approximations on partitions learned by deep neural networks. The architecture is designed to play to the strengths of each approach, while still achieving good convergence rates as demonstrated in the results.
Part B: Seminar
A Layer-Parallel Approach for Training Deep Neural Networks
Deep neural networks are a powerful machine learning tool with the capacity to “learn” complex nonlinear relationships described by large data sets. Despite their success training these models remains a challenging and computationally intensive undertaking. In this talk we will present a layer-parallel training algorithm that exploits a multigrid scheme to accelerate both forward and backward propagation. Introducing a parallel decomposition between layers requires inexact propagation of the neural network. The multigrid method used in this approach stitches these subdomains together with sufficient accuracy to ensure rapid convergence. We demonstrate an order of magnitude wall-clock time speedup over the serial approach, opening a new avenue for parallelism that is complementary to existing approaches. In a more recent development, we discuss applying the layer-parallel methodology to recurrent neural networks. In particular, we study the generalized recurrent unit (GRU) architecture. We demonstrate its relation to a simple ODE formulation that facilitates application of the layer-parallel approach. Results are demonstrating performance improvements on a human activity recognition (HAR) data set are presented.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Prof. Misha Siddhartha
professor at the Department of Mathematics, ETH Zurich.
Part A:Lecture
Deep Learning and Computations of PDEs.
Abstract: In this talk, we will review some recent results on computing PDEs with deep neural networks. The focus will be on the design of neural networks as fast and efficient surrogates for PDEs. We will start with parametric PDEs and talk about novel modifications that enable standard deep neural networks to provide efficient PDE surrogates and apply them for prediction, uncertainty quantification and PDE constrained optimization. We will also cover very recent results on operator regression using novel architectures such as DeepOnets and Fourier Neural Operators, and their application to PDEs.
Part B: Seminar
On Physical Informed Neural Networks (PINNs) for computing PDEs.
Abstract: We will describe PINNs and illustrate several examples for using PINNs for solving PDEs. Our aim would be to elucidate mechanisms that underpin the success of PINNs in approximating classical solutions to PDEs by deriving bounds on the resulting error. Examples of a variety of linear and nonlinear PDEs will be provided including PDEs in high dimensions and inverse problems for PDEs.
MATERIAL
You can download the content of the lectures here
pwd is pinn
Venue
Classes will take place online
Registration
Please, send an email to
Organizers:
Rolf Krause, Maria Nestola, Patrick Zulian, Alena Kopaničáková., Simone Pezzuto, Luca Gambardella