Speaker
Description
Feed-forward neural networks can be interpreted as dyscrete switched dynamical systems where the switching rule is determined by the training process. The stability of switched systems has been extensively investigated in the linear case, where the key ingredient is given by the joint spectral radius of the family of matrices that determines the dynamics of the system. On the other hand, neural networks used in modern deep learning typically require nonlinear subhomogeneous activation functions for which new theoretical investigation is needed. Thus, we investigate switched systems that alternate maps from a (possibly infinite) class $\mathcal{F}$ of nonlinear functions, following an unknown switching rule. In particular we focus on families of sub-homogeneous nonlinear functions that preserve the ordering induced by a cone. We introduce the notion of nonlinear joint spectral radius (JSR) of a family of such functions. Then we show that the JSR provides information about the stability of the system. After that, we investigate properties of the nonlinear JSR, including a dual formulation in terms of monotone prenorms of the functions in $\mathcal{F}$, and a formulation in terms of the spectral radii of the functions in the semigroup generated by $\mathcal{F}$. Finally, making use of the different formulations of the JSR, we present an algorithm devoted to the computation of the nonlinear JSR. Our algorithm, inspired by the polytopal algorithm used in the linear case, iteratively builds a monotone extremal prenorm for the system in terms of the Minkowski functional of a finitely generated subset of the cone.
