Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

We investigate the use of multiple transmitting and/or receiving antennas for single user communications over the additive Gaussian channel with and without fading. We derive formulas for the capacities and error exponents of such channels, and describe computational procedures to evaluate such formulas. We show that the potential gains of such multi-antenna systems over single-antenna systems is rather large under independenceassumptions for the fades and noises at different receiving antennas.

/pdf/capacity-of-multi-antenna-gaussian-channels-3axdfdd5ns.pdf

Capacity of Multi‐antenna Gaussian Channels

This paper considers a natural error correcting problem with real valued input/output. We wish to recover an input vector f/spl isin/R/sup n/ from corrupted measurements y=Af+e. Here, A is an m by n (coding) matrix and e is an arbitrary and unknown vector of errors. Is it possible to recover f exactly from the data y? We prove that under suitable conditions on the coding matrix A, the input f is the unique solution to the /spl lscr//sub 1/-minimization problem (/spl par/x/spl par//sub /spl lscr/1/:=/spl Sigma//sub i/|x/sub i/|) min(g/spl isin/R/sup n/) /spl par/y - Ag/spl par//sub /spl lscr/1/ provided that the support of the vector of errors is not too large, /spl par/e/spl par//sub /spl lscr/0/:=|{i:e/sub i/ /spl ne/ 0}|/spl les//spl rho//spl middot/m for some /spl rho/>0. In short, f can be recovered exactly by solving a simple convex optimization problem (which one can recast as a linear program). In addition, numerical experiments suggest that this recovery procedure works unreasonably well; f is recovered exactly even in situations where a significant fraction of the output is corrupted. This work is related to the problem of finding sparse solutions to vastly underdetermined systems of linear equations. There are also significant connections with the problem of recovering signals from highly incomplete measurements. In fact, the results introduced in this paper improve on our earlier work. Finally, underlying the success of /spl lscr//sub 1/ is a crucial property we call the uniform uncertainty principle that we shall describe in detail.

/pdf/decoding-by-linear-programming-4q5aw83zqx.pdf

Decoding by linear programming

This paper addresses digital communication in a Rayleigh fading environment when the channel characteristic is unknown at the transmitter but is known (tracked) at the receiver. Inventing a codec architecture that can realize a significant portion of the great capacity promised by information theory is essential to a standout long-term position in highly competitive arenas like fixed and indoor wireless. Use (n T , n R ) to express the number of antenna elements at the transmitter and receiver. An (n, n) analysis shows that despite the n received waves interfering randomly, capacity grows linearly with n and is enormous. With n = 8 at 1% outage and 21-dB average SNR at each receiving element, 42 b/s/Hz is achieved. The capacity is more than 40 times that of a (1, 1) system at the same total radiated transmitter power and bandwidth. Moreover, in some applications, n could be much larger than 8. In striving for significant fractions of such huge capacities, the question arises: Can one construct an (n, n) system whose capacity scales linearly with n, using as building blocks n separately coded one-dimensional (1-D) subsystems of equal capacity? With the aim of leveraging the already highly developed 1-D codec technology, this paper reports just such an invention. In this new architecture, signals are layered in space and time as suggested by a tight capacity bound.

http://www.ece.ualberta.ca/~hcdc/Library/StCommClass/Fosc96.pdf

Layered space-time architecture for wireless communication in a fading environment when using multi-element antennas

Suppose we are given a vector f in a class FsubeRopfN , e.g., a class of digital signals or digital images. How many linear measurements do we need to make about f to be able to recover f to within precision epsi in the Euclidean (lscr2) metric? This paper shows that if the objects of interest are sparse in a fixed basis or compressible, then it is possible to reconstruct f to within very high accuracy from a small number of random measurements by solving a simple linear program. More precisely, suppose that the nth largest entry of the vector |f| (or of its coefficients in a fixed basis) obeys |f|(n)lesRmiddotn-1p/, where R>0 and p>0. Suppose that we take measurements yk=langf# ,Xkrang,k=1,...,K, where the Xk are N-dimensional Gaussian vectors with independent standard normal entries. Then for each f obeying the decay estimate above for some 0<p<1 and with overwhelming probability, our reconstruction ft, defined as the solution to the constraints yk=langf# ,Xkrang with minimal lscr1 norm, obeys parf-f#parlscr2lesCp middotRmiddot(K/logN)-r, r=1/p-1/2. There is a sense in which this result is optimal; it is generally impossible to obtain a higher accuracy from any set of K measurements whatsoever. The methodology extends to various other random measurement ensembles; for example, we show that similar results hold if one observes a few randomly sampled Fourier coefficients of f. In fact, the results are quite general and require only two hypotheses on the measurement ensemble which are detailed

/pdf/near-optimal-signal-recovery-from-random-projections-2mmu9nt75f.pdf

Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies?

Wigner Matrices and Semicircular Law.- Sample Covariance Matrices and the Mar#x010D enko-Pastur Law.- Product of Two Random Matrices.- Limits of Extreme Eigenvalues.- Spectrum Separation.- Semicircular Law for Hadamard Products.- Convergence Rates of ESD.- CLT for Linear Spectral Statistics.- Eigenvectors of Sample Covariance Matrices.- Circular Law.- Some Applications of RMT.

Spectral Analysis of Large Dimensional Random Matrices

A previously proposed model for memory based on neurophysiolo gical considerations is reviewed. We assume that (a) nervous system activity is usefully represented as the set of simultaneous individual neuron activities in a group of neurons; (b) different memory traces make use of the same synapses; and (c) synapses associate two patterns of neural activity by incrementing synaptic connectivity proportionally to the product of pre- and postsynaptic activity, forming a matrix of synaptic connectivities. We extend this model by (a) introducing positive feedback of a set of neurons onto itself and (b) allowing the individual neurons to saturate. A hybrid model, partly analog and partly binary, arises. The system has certain characteristics reminiscent of analysis by distinctive features. Next, we apply the model to "categorical perception." Finally, we discuss probability learning. The model can predict overshooting, recency data, and probabilities occurring in systems with more than two events with reasonably good accuracy.

Distinctive Features, Categorical Perception, and Probability Learning: Some Applications of a Neural Model.

On the empirical distribution of eigenvalues of a class of large dimensional random matrices

http://www.math.lsa.umich.edu/~baik/Publication_files/paper17.pdf

Eigenvalues of large sample covariance matrices of spiked population models

Let $B_n = (1/N)T_n^{1/2}X_n X_n^* T_n^{1/2}$, where $X_n$ is $n \times N$ with i.i.d. complex standardized entries having finite fourth moment and $T_n^{1/2}$ is a Hermitian square root of the nonnegative definite Hermitian matrix $T_n$. It is known that, as $n \to \infty$, if $n/N$ converges to a positive number and the empirical distribution of the eigenvalues of $T_n$ converges to a proper probability distribution, then the empirical distribution of the eigenvalues of $B_n$ converges a.s. to a nonrandom limit. In this paper we prove that, under certain conditions on the eigenvalues of $T_n$, for any closed interval outside the support of the limit, with probability 1 there will be no eigenvalues in this interval for all $n$ sufficiently large.

/pdf/no-eigenvalues-outside-the-support-of-the-limiting-spectral-2cpf7510mw.pdf

Jack W. Silverstein

Papers

Spectral Analysis of Large Dimensional Random Matrices

Distinctive Features, Categorical Perception, and Probability Learning: Some Applications of a Neural Model.

On the empirical distribution of eigenvalues of a class of large dimensional random matrices

Eigenvalues of large sample covariance matrices of spiked population models

No eigenvalues outside the support of the limiting spectral distribution of large-dimensional sample covariance matrices