FOMICS-DADSI SEMINARS ON SCIENTIFIC LEARNING

Simulations in Science and Engineering" is pleased to announce a Winter School for PhD
and prospective/current/enrolled master students in the fields of modern mathematics and computational science:
 
Scientific Learning
 

dadsilogo

 
 

The Institute of Computational Science (ICS) at the Università della Svizzera italiana (USI) announces six seminars about  Machine Learning, Neural Networks, Physics Informed Neural Networks
The seminars are supported and organized by the Swiss Graduate School FOMICS - "Foundations in Mathematics and Informatics for Computer Simulations in Science and Engineering”, which is located at Euler Institute
  
Topics

 Machine Learning, Neural Networks, Physics Informed Neural Networks

 

 

  PROGRAM 
 Time Speaker & Place

May 19 

5:00-7:00 pm

UTC+2:00 Zurich time

Prof. Paris Perdikaris

Lecture: 5:00-6:00 pm 

Seminar: 6:00-7:00 pm

June 2

5:00-7:00 pm

UTC+2:00 Zurich time

Prof. Jonathan Siegel

Lecture: 5:00-6:00 pm 

Seminar: 6:00-7:00 pm

June 16

5:00-7:00 pm

UTC+2:00 Zurich time

Dr. Eric Cyr

Lecture: 5:00-6:00 pm 

Seminar: 6:00-7:00 pm

June 23

5:00-7:00 pm

UTC+2:00 Zurich time

Prof. Mishra Siddhartha

Lecture: 5:00-6:00 pm 

Seminar: 6:00-7:00 pm

 
 
 
 
List of Speakers 

 

Prof. Paris Perdikaris

assistant professor at Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania.

 

 

 

Part A:Lecture

Making neural networks physics-informed
 
Leveraging advances in automatic differentiation, physics-informed neural networks are introducing a new paradigm in tackling forward and inverse problems in computational mechanics. Under this emerging paradigm, unknown quantities of interest are typically parametrized by deep neural networks, and a multi-task learning problem is posed with the dual goal of fitting observational data and approximately satisfying a given physical law, mathematically expressed via systems of partial differential equations (PDEs). PINNs have demonstrated remarkable flexibility across diverse applications, but, despite some empirical success, a concrete mathematical understanding of the mechanisms that render such constrained neural network models effective is still lacking. In fact, more often than not, PINNs are notoriously hard to train, especially for forward problems exhibiting high-frequency or multi-scale behavior. In this talk we will discuss the basic principles of making neural networks physics informed with an emphasis on the caveats one should be aware of and how those can be addressed in practice.
 
Part B: Seminar
Learning the solution operator of parametric partial differential equations with physics-informed DeepONets
 
Deep operator networks (DeepONets) are receiving increased  attention thanks to their demonstrated capability to approximate  nonlinear operators between infinite-dimensional Banach spaces. However, despite their remarkable early promise, they typically require large training data-sets consisting of paired input-output observations which may be expensive to obtain, while their predictions may not be consistent with the underlying physical principles that generated the observed data. In this work,  we propose a novel model class coined as physics-informed DeepONets,  which introduces an effective regularization mechanism for biasing the outputs of DeepOnet models towards ensuring physical consistency. This is accomplished by leveraging automatic differentiation to impose the underlying physical laws via soft penalty constraints during model training. We demonstrate that this simple, yet remarkably effective extension can not only yield a significant improvement in the predictive accuracy of DeepOnets, but also greatly reduce the need for large training data-sets. To this end, a remarkable observation is that physics-informed DeepONets are capable of solving parametric partial differential equations (PDEs) without any paired input-output observations, except for a set of given initial or boundary conditions. We illustrate the effectiveness of the proposed framework through a series of comprehensive numerical studies across various types of PDEs.  Strikingly, a trained physics informed DeepOnet model can predict the solution of O(1000)  time-dependent PDEs in a fraction of a second -- up to three orders of magnitude faster compared to a conventional PDE solver.
 
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
Prof. Jonathan Siegel
assistant professor at the Department of Mathematics, Penn State.
 
 
 Part A:Lecture
 
The Approximation Theory of Neural Networks
 
An important component in understanding the potential and limitations of neural networks is a study of how efficiently neural networks can approximate a given target function class. Classical results which we will cover include the universal approximation property of neural networks and the theory of Barron and Jones which gives quantitative approximation rates for shallow neural networks. We will also discuss approximation rates for deep ReLU networks and their relationship with finite elements methods. Finally, we will introduce the notions of metric entropy and  n-widths which are fundamental in approximation theory and permit a comparison between neural networks and traditional methods of approximation.
 
 
 
Part B: Seminar
 
The Metric Entropy and n-widths of Shallow Neural Networks
 
The metric entropy and n-widths are fundamental quantities in approximation theory which control the fundamental limits of linear and non-linear approximation and statistical estimation for a given class of functions. In this talk, we will derive the spaces of functions which  can be efficiently approximated by shallow neural networks for a wide variety of activation functions, and we will calculate the metric  entropy and n-widths for these spaces. Consequences of these results include: the optimal approximation rates which can be attained  for shallow neural networks, that shallow neural networks dramatically outperform linear methods of approximation, and indeed that  shallow neural networks outperform all stable methods of approximation on these spacesFinally, we will discuss insights into how neural networks break the curse of dimensionality.
 
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
Dr. Eric Cyr
reasercher at the Computational Mathematics Department, Sandia National Laboratories
 
Part A:Lecture
 
An Adaptive Basis Perspective to Improve Initialization and Accelerate Training of DNNs
 
Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We provide analysis of these techniques and illustrate via numerical examples dramatic increases in accuracy and convergence rate for benchmarks characterizing scientific applications where DNNs are currently used, including regression problems and physics-informed neural networks for the solution of partial differential equations. In addition, we present a partition of unity (POU) architecture capable of achieving hp like convergence. This methodology, introduces traditional polynomial approximations on partitions learned by deep neural networks. The architecture is designed to play to the strengths of each approach, while still achieving good convergence rates as demonstrated in the results.
 
 
Part B: Seminar 
 
A Layer-Parallel Approach for Training Deep Neural Networks
 
Deep neural networks are a powerful machine learning tool with the capacity to “learn” complex nonlinear relationships described by large data sets. Despite their success training these models remains a challenging and computationally intensive undertaking. In this talk we will present a layer-parallel training algorithm that exploits a multigrid scheme to accelerate both forward and backward propagation. Introducing a parallel decomposition between layers requires inexact propagation of the neural network. The multigrid method used in this approach stitches these subdomains together with sufficient accuracy to ensure rapid convergence. We demonstrate an order of magnitude wall-clock time speedup over the serial approach, opening a new avenue for parallelism that is complementary to existing approaches. In a more recent development, we discuss applying the layer-parallel methodology to recurrent neural networks. In particular, we study the generalized recurrent unit (GRU) architecture. We demonstrate its relation to a simple ODE formulation that facilitates application of the layer-parallel approach. Results are demonstrating performance improvements on a human activity recognition (HAR) data set are presented.
 
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 
Prof. Misha Siddhartha
professor at the Department of Mathematics,  ETH Zurich.
 
 
Part A:Lecture
 
Deep Learning and Computations of PDEs.

 

Abstract: In this talk, we will review some recent results on computing PDEs with deep neural networks. The focus will be on the design of neural networks as fast and efficient surrogates for PDEs. We will start with parametric PDEs and talk about novel modifications that enable standard deep neural networks to provide efficient PDE surrogates and apply them for prediction, uncertainty quantification and PDE constrained optimization. We will also cover very recent results on operator regression using novel architectures such as DeepOnets and Fourier Neural Operators, and their application to PDEs. 

 

Part B: Seminar 

 

On Physical Informed Neural Networks (PINNs) for computing PDEs. 

 

Abstract: We will describe PINNs and illustrate several examples for using PINNs for solving PDEs. Our aim would be to elucidate mechanisms that underpin the success of PINNs in approximating classical solutions to PDEs  by deriving bounds on the resulting error. Examples of a variety of linear and nonlinear PDEs will be provided including PDEs in high dimensions and inverse problems for PDEs. 

 
MATERIAL 

You can download the content of the lectures here

https://owncloud.ics.usi.ch/owncloud/index.php/s/4spzesMOsEO35GP

pwd is pinn

 

 
 
 
 
Venue
Classes will take place online 
 
Registration
Please, send an email to
 
Organizers:
Rolf Krause,   Maria Nestola,  Patrick Zulian, Alena Kopaničáková.,  Simone Pezzuto, Luca Gambardella

cardio-centro-ticnic-logo

logo cscs