A. Balavoine, J. Romberg, and C.J. Rozell.
Convergence and rate analysis of neural networks for sparse
approximation.
IEEE Transactions on Neural Networks and Learning Systems,
2012.
In press. Revision posted March 27, 2012.
[ .pdf ]
We present an analysis of the Locally Competitive Algorithm (LCA), a Hopfield-style neural network that efficiently solves sparse approximation problems (e.g., approximating a vector from a dictionary using just a few non-zero coefficients). This class of problems plays a significant role in both theories of neural coding and applications in signal processing. However, the LCA lacks analysis of its convergence properties and previous results on neural networks for nonsmooth optimization do not apply to the specifics of the LCA architecture. We show that the LCA has desirable convergence properties, such as stability and global convergence to the optimum of the objective function when it is unique. Under some mild conditions, the support of the solution is also proven to be reached in finite time. Furthermore, some restrictions on the problem specifics allow us to characterize the convergence rate of the system by showing that the LCA converges exponentially fast with an analytically bounded convergence rate. We support our analysis with several illustrative simulations.
S. Shapero, C. Rozell, and P. Hasler.
Low power sparse approximation on reconfigurable analog hardware.
2012.
In review. Submitted November 2011.
Compressed sensing is an important optimization problem in signal and image processing applications. A Hopfield-Network-like analog system is proposed as a solution, using the Locally Competitive Algorithm (LCA) to solve an overcomplete l1 sparse approximation problem. A scalable system architecture using sub-threshold currents is described, including vector matrix multipliers (VMMs) and a nonlinear thresholder. A 4x6 nonlinear system is implemented on the RASP 2.9v chip, a Field Programmable Analog Array with directly programmable floating gate elements, allowing highly accurate VMMs. The circuit successfully reproduced the outputs of a digital optimization program, converging to within 4.8% RMS, and an optimization cost only 1.3% higher on average. The active circuit consumed 29μA of current at 2.4V, and converges on solutions in 240μs. A smaller 2x3 system is also implemented. Extrapolating the scaling trends to N=1000 node system, the Analog LCA compares favorably with State-of-the-Art digital solutions.
M. Zhu and C.J. Rozell.
Visual nonclassical receptive field effects emerge from sparse coding
in a dynamical system.
2012.
In review. Submitted May 2012.
Extensive electrophysiology studies have shown that many V1 simple cells receive contextual influence, where stimuli not part of the classical receptive field (CRF) can modulate the cell’s response to CRF stimuli. Mechanistic models of these non-classical receptive field (nCRF) effects do not address the role of these effects in the optimal coding or processing of natural stimuli. We demonstrate that a wide variety of nCRF effects are emergent properties of a single sparse coding model implemented in a neurally plausible network structure (requiring no tuning of any parameters to produce the various effects). In particular, this model produces both individual cells showing canonical nCRF effects, as well as the population diversity reported in the physiology literature. These results show that the sparse coding hypothesis, when coupled with a biophysically plausible implementation, can provide apparently disparate response properties with a unified interpretation based on the efficient coding of natural scenes.
A.A. Kressner, D.V. Anderson, and C.J. Rozell.
Evaluating the generalization of the hearing aid speech quality index
(HASQI).
2012.
In review. Submitted April 2012.
Many developers of audio signal processing strategies rely on objective measures of quality for initial evaluations of algorithms. As such, objective measures should be robust, and they should be able to predict quality accurately regardless of the dataset or testing conditions. Kates and Arehart have developed the Hearing Aid Speech Quality Index (HASQI) to predict the effects of noise, nonlinear distortion, and linear filtering on speech quality for both normal-hearing and hearing-impaired listeners, and they report very high performance with their training and testing datasets [Kates, J. and Arehart, K., Audio Eng. Soc., 58(5), 363-381 (2010)]. In order to investigate the generalizability of HASQI, we test its ability to predict normal-hearing listeners’ subjective quality ratings of a dataset on which it was not trained. This dataset is designed specifically to contain a wide range of distortions introduced by real-world noises which have been processed by some of the most common noise suppression algorithms in hearing aids. We show that HASQI achieves prediction performance comparable to PESQ, the standard for objective measures of quality, as well as some of the other measures in the literature. Furthermore, we identify areas of weakness and show that training can improve quantitative prediction.
A.S. Charles, P. Garrigues, and C.J. Rozell.
A common network architecture can efficiently implement diverse
sparsity-based inference problems.
2012.
In review. Submitted May 2012.
Recent signal processing developments have greatly expanded the range of sparsity based optimization functions for a variety of statistical models. Initial work in finding neurally plausible systems resulted in the locally competitive algorithms for solving basic sparsity inducing optimizations. Incorporating these new model types, however, would greatly increase the relevance of neural systems for solving regularized least-squares problems. In this work we show analytically that a wide range of sparse approximation problems can be solved in the LCA architecture, including approximate lp norms, modified l1 norms, re-weighted l1 and l2, the block l1 norm and classic Tikhonov regularization.
A.S. Charles, H.L. Yap, and C.J.
Rozell.
Short term network memory capacity via the restricted isometry
property.
2012.
In review. Submitted May 2012.
The transient activity in a recurrent (i.e., feedback) network due to past input perturbations can be viewed as a type of short-term memory (STM) where the collective network state at a given time can be used to recover a past input sequence. Past analyses (generally using Gaussian input statistics) have shown STM capacities of linear recurrent networks that are limited to the number of nodes in the network.
We leverage recent results in the emerging field of compressed sensing to provide rigorous, non-asymptotic recovery guarantees for STM in networks when the inputs can be sparsely represented in a basis. These results illustrate that network STM capacities can be significantly higher than the number of nodes in the network. We provide both perfect recovery guarantees for finite inputs, as well as results on the recovery tradeoffs when the network has an infinitely long input sequence. The latter analysis highlights the fact that when a network is faced with an infinitely long streaming input, the system has an optimal input recovery length that balances errors due to omission and recall mistakes.
J.Y. Park, H.L. Yap, C.J. Rozell, and M.B. Wakin.
Concentration of measure for block diagonal matrices with
applications to compressive signal processing.
IEEE Transactions on Signal Processing, 59:5859-5875, December
2011.
[ .pdf ]
Theoretical analysis of randomized, compressive operators often depends on a concentration of measure inequality for the operator in question. Typically, such inequalities quantify the likelihood that a random matrix will preserve the norm of a signal after multiplication. Concentration of measure results are well-established for unstructured compressive matrices, populated with independent and identically distributed (i.i.d.) random entries. Many real-world acquisition systems, however, are subject to architectural constraints that make such matrices impractical. In this paper we derive concentration of measure bounds for two types of block diagonal compressive matrices, one in which the blocks along the main diagonal are random and independent, and one in which the blocks are random but equal. For both types of matrices, we show that the likelihood of norm preservation depends on certain properties of the signal being measured, but that for the best case signals, both types of block diagonal matrices can offer concentration performance on par with their unstructured, i.i.d. counterparts. We support our theoretical results with illustrative simulations as well as analytical and empirical investigations of several signal classes that are highly amenable to measurement using block diagonal matrices. We also discuss applications of these results in ensuring stable embeddings for various signal families and in establishing performance guarantees for
solving various signal processing tasks (such as detection and classification) directly in the compressed domain.
H.L. Yap and C.J. Rozell.
Stable Takens' embeddings for linear dynamical systems.
IEEE Transactions on Signal Processing, 59(10):4781-4794,
October 2011.
[ .pdf ]
Takens' Embedding Theorem remarkably established that concatenating M previous outputs of a dynamical system into a vector (called a delay coordinate map) can be a one-to-one mapping of a low-dimensional attractor from the system state-space.
However, Takens' theorem is fragile because even small imperfections can induce arbitrarily large errors in the attractor representation. We extend Takens' result to establish explicit, non-asymptotic sufficient conditions for a delay coordinate map to form a stable embedding in the restricted case of linear dynamical systems and observation functions.
Our work is inspired by the field of Compressive Sensing (CS), where results guarantee that low-dimensional signal families can be robustly reconstructed if they are stably embedded by a measurement operator.
However, in contrast to typical CS results, i) our sufficient conditions are independent of the size of the ambient state space (N), and ii) some system and measurement pairs have fundamental limits on the conditioning of the embedding (i.e., how close it is to an isometry), meaning that further measurements beyond some point add no further significant value.
We use several simple simulations to explore the conditions of the main results, including the tightness of the bounds and the convergence speed of the stable embedding.
A.S. Charles, B.A. Olshausen, and C.J. Rozell.
Learning sparse codes for hyperspectral imagery.
IEEE Journal of Selected Topics in Signal Processing,
5(5):963-978, September 2011.
[ .pdf ]
The spectral features in hyperspectral imagery (HSI) contain significant structure that, if properly characterized could enable more efficient data acquisition and improved data analysis. Because most pixels contain reflectances of just a few materials, we propose that a sparse coding model is well-matched to HSI data. Sparsity models consider each pixel as a combination of just a few elements from a larger dictionary, and this approach has proven effective in a wide range of applications. Furthermore, previous work has shown that optimal sparse coding dictionaries can be learned from a dataset with no other a priori information (in contrast to many HSI “endmember” discovery algorithms that assume the presence of pure spectra or side information).
We modified an existing unsupervised learning approach and applied it to HSI data (with significant ground truth labeling) to learn an optimal sparse coding dictionary. Using this learned dictionary, we demonstrate three main findings: i) the sparse coding model learns spectral signatures of materials in the scene and locally approximates nonlinear manifolds for individual materials, ii) this learned dictionary can be used to infer HSI-resolution data with very high accuracy from simulated imagery collected at multispectral-level resolution, and iii) this learned dictionary improves the performance of a supervised classification algorithm, both in terms of the classifier complexity and generalization from very small training sets.
C.J. Rozell, D.H Johnson, R.G. Baraniuk, and B.A. Olshausen.
Sparse coding via thresholding and local competition in neural
circuits.
Neural Computation, 20(10):2526-2563, October 2008.
[ .pdf ]
While evidence indicates that neural systems may be
employing sparse approximations to represent sensed
stimuli, the mechanisms underlying this ability are
not understood. We describe a local ly competitive
algorithm (LCA) that solves a collection of sparse
coding principles minimizing a weighted combination
of mean-squared error (MSE) and a coefficient cost
function. LCAs are designed to be implemented in a
dynamical system composed of many neuron-like
elements operating in parallel. These algorithms use
thresholding functions to induce local (usually
one-way) inhibitory competitions between nodes to
produce sparse representations. LCAs produce
coefficients with sparsity levels comparable to the
most popular centralized sparse coding algorithms
while being readily suited for neural
implementation. Addi- tionally, LCA coefficients for
video sequences demonstrate inertial properties that
are both qualitatively and quantitatively more
regular (i.e., smoother and more predictable) than
the coefficients produced by greedy algorithms.
We show that an Au nanoshell with a pH sensitive
molecular adsorbate functions as a standalone, all-optical nanoscale
pH meter that monitors its local environment through the
pH-dependent surface enhanced Raman scattering (SERS) spectra of the
adsorbate molecules. Moreover, we also show how the performance of
such a functional nanodevice can be quantitatively assessed. The
complex spectral output is reduced to a simple device characteristic
by application of a locally linear manifold approximation
algorithm. The average accuracy of the nano-“meter” was found to
be ± 0.10 pH units across its operating range.
C.J. Rozell and D.H. Johnson.
Analyzing the robustness of redundant population codes in sensory and
feature extraction systems.
Neurocomputing, 69(10-12):1215-1218, June 2006.
[ .pdf ]
Sensory systems often use groups of redundant neurons to represent
stimulus information both during transduction and population coding of
features. This redundancy makes the system more robust to corruption
in the representation. We approximate neural coding as a projection of
the stimulus onto a set of vectors, with the result encoded by spike
trains. We use the formalism of frame theory to quantify the inherent
noise reduction properties of such population codes. Additionally,
computing features from the stimulus signal can also be thought of as
projecting the coefficients of a sensory representation onto another
set of vectors specific to the feature of interest. The conditions
under which a combination of different features form a complete
representation for the stimulus signal can be found through a recent
extension to frame theory called “frames of subspaces.” We extend
the frame of subspaces theory to quantify the noise reduction
properties of a collection of redundant feature spaces.
C.J. Rozell and D.H. Johnson.
Examining methods for estimating mutual information in spiking neural
systems.
Neurocomputing, 65-66C:429-434, June 2005.
[ .pdf ]
Mutual information enjoys wide use in the computational neuroscience community for
analyzing spiking neural systems. Its direct calculation is difficult
because estimating the joint stimulus-response distribution requires a
prohibitive amount of data. Consequently, several techniques have
appeared for bounding mutual information that rely on less data. We
examine two upper bound techniques and find that they are either
unreliable or introduce strong assumptions about the neural code. We also
examine two lower bounds, showing that they can be very loose and
possibly bear little relation to the mutual information's actual value.
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Measuring information transfer in crayfish sustaining fiber spike
generators.
Biological Cybernetics, 90(2):89-97, February 2004.
[ .pdf ]
We present a method based
on information-theoretic distances for measuring the information
transfer efficiency of voltage to impulse encoders. In response to light
pulses, we simultaneously recorded the EPSP and spiking output of
crayfish sustaining fibers. To measure the distance between analog
EPSP responses, we developed a membrane noise model that accurately
captures stimulus-induced nonstationarities. By comparing the EPSP
and spike responses, we found encoding efficiencies on the order
of 10-4, with interesting dynamics occurring during initial
transients. A simple analog to point-process converter predicted the small
information transfer efficiencies and dynamic properties we measured.
Copyright held by Springer-Verlag. The original
publication is available at springerlink.com. DOI:10.1007/s00422-003-0458-y
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Information processing during transient responses in the crayfish
visual system.
Neurocomputing, 52-54:53-58, June 2003.
[ .pdf ]
We analyzed sustaining fiber
responses in the crayfish visual system to light pulses using information
processing techniques. The light pulse stimuli elicited a transient and
a steady-state component in the EPSP input and in the firing rate
of the spike train output. The overall information transfer of the
system was very low (10-4), with a sharp increase during the
transient portion of the response followed by a steady decrease. The
information transfer dynamics are consistent with a simple spike
generator model that depends explicitly on stimulus changes. The
present analysis also corroborates the observed light reflex behavior.
D.H. Johnson, I.N. Goodman, and C.J. Rozell.
Information theory and systems neuroscience.
In S. Grün and S. Rotter, editors, Analysis of parallel
spike trains. Springer-Verlag, 2010.
H.L. Yap, A. Charles, and C.J.
Rozell.
The restricted isometry property for echo state networks with
applications to sequence memory capacity.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
D. Sale, C.J. Rozell, J.K. Romberg, and A.D. Lanterman.
Compressive LADAR in realistic environments.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
S. Shapero, C. Rozell, A. Balavoine, and P. Hasler.
A scalable implementation of sparse approximation on a Field
Programmable Analog Array.
In IEEE Biomedical Circuits and Systems Conference (BioCAS), La
Jolla, CA, November 2011.
A. Kressner, D. Anderson, and C. Rozell.
Robustness of the hearing aid speech quality index (HASQI).
In IEEE Workshop on Applications of Signal Processing to Audio
and Acoustics (WASPAA), New Paltz, NY, October 2011.
[ .pdf ]
Objective measures of speech quality have been the subject of significant prior work, particularly in the areas of speech codecs and communication channels for normal-hearing listeners. One of the primary concerns of researchers in this area is how these metrics generalize to datasets or listener studies which are “unknown” to the measures. Another growing concern is how these metrics perform for the hearing-impaired community. Researchers working with the this community need to be able to predict how hearing-impaired listeners will perceive the quality of speech, as well as how they will perceive the quality of speech processed specifically by hear- ing aids. A relatively recent metric, the Hearing Aid Speech Quality Index (HASQI), is a model-based objective measure of quality developed in the context of hearing aids for normal-hearing and hearing-impaired listeners (Kates & Arehart, Journal of the Audio Engineering Society, 2010). As such, HASQI makes substantial progress on some of the generalization issues. However, HASQI has not been tested thus far on any datasets other than the one on which it was trained. The objective of this study is to demonstrate the robustness of HASQI in predicting subjective quality. We use an “unknown” dataset of noisy speech processed by noise suppression algorithms, along with a corresponding set of subjective quality scores from normal-hearing listeners, to demonstrate HASQI’s prediction performance. Furthermore, we compare HASQI’s performance with that of several other objective measures in order to provide a point of reference.
M.S. Asif, A. Charles, J. Romberg, and C. Rozell.
Estimation and dynamic updating of time-varying signals with sparse
variations.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Prague, Czech Republic, May 2011.
[ .pdf ]
This paper presents an algorithm for an l1-regularized Kalman filter. Given observations of a discrete-time linear dynamical system with sparse errors in the state evolution, we estimate the state sequence by solving an optimization algorithm that balances fidelity to the measurements (measured by the standard l2 norm) against the sparsity of the innovations (measured using the l1 norm). We also derive an efficient algorithm for updating the estimate as the system evolves. This dynamic updating algorithm uses a homotopy scheme that tracks the solution as new measurements are slowly worked into the system and old measurements are slowly removed. The effective cost of adding new measurements is a number of low-rank updates to the solution of a linear system of equations that is roughly proportional to the joint sparsity of all the innovations in the time interval of interest.
H.L. Yap, M.B. Wakin, and C.J. Rozell.
Stable manifold embeddings with operators satisfying the restricted
isometry property.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
Signals of interests can often be thought to come from a low dimensional signal model.
The exploitation of this fact has led to many recent interesting advances in signal processing, one notable example being in the field of compressive sensing (CS).
The literature on CS has established that many matrices satisfy the Restricted Isometry Property (RIP), which guarantees a stable (i.e., distance-preserving) embedding of a sparse signal model from an undersampled linear measurement system.
In this work, we study the stable embedding of manifold signal models using matrices that satisfy the RIP.
We show that by paying reasonable additional factors in the number of measurements, all matrices that satisfy the RIP can also be used (in conjunction with a random sign sequence) to obtain a stable embedding of a manifold.
H.L. Yap, A. Eftekhari, M.B. Wakin, and C.J. Rozell.
The restricted isometry property for block diagonal matrices.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In compressive sensing (CS), the Restricted Isometry Property (RIP) is a powerful condition on measurement operators which ensures robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms.
Early papers in CS showed that Gaussian random matrices satisfy the RIP with high probability, but such matrices are usually undesirable in practical applications due to storage limitations, computational considerations, or the mismatch of such matrices with certain measurement architectures.
To alleviate some or all of these difficulties, recent research efforts have focused on structured random matrices.
In this paper, we study block diagonal measurement matrices where each block on the main diagonal is itself a Gaussian random matrix.
The main result of this paper shows that such matrices can indeed satisfy the RIP but that the requisite number of measurements depends on the coherence of the basis in which the signals are sparse.
In the best case-for signals that are sparse in the frequency domain-these matrices perform nearly as well as dense Gaussian random matrices despite having many fewer nonzero entries.
A. Charles, M.S. Asif, J. Romberg, and C. Rozell.
Sparsity penalties in dynamical system estimation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In this work we address the problem of state estimation in dynamical systems using recent developments in compressive sensing and sparse approximation. We formulate the traditional Kalman filter as a one-step update optimization procedure which leads us to a more unified framework, useful for incorporating sparsity constraints. We introduce three combinations of two sparsity conditions (sparsity in the state and sparsity in the innovations) and write recursive optimization programs to estimate the state for each model. This paper is meant as an overview of different methods for incorporating sparsity into the dynamic model, a presentation of algorithms that unify the support and coefficient estimation, and a demonstration that these suboptimal schemes can actually show some performance improvements (either in estimation error or convergence time) over standard optimal methods that use an impoverished model.
A. Balavoine, C.J. Rozell, and J.K. Romberg.
Global convergence of the Locally Competitive Algorithm.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
The Locally Competitive Algorithm (LCA) is a continuous-time dynamical system designed to solve the problem of sparse approximation. This class of approximation problems plays an important role in producing state-of-the-art results in many signal processing and inverse problems, and implementing the LCA in analog VLSI may significantly improve the time and power necessary to solve these optimization programs. The goal of this paper is to analyze the dynamical behavior of the LCA system and guarantee its convergence and stability. We show that the LCA converges from any initial point. We also show that fixed points of the system are extrema of the sparse approximation objective function when designed for a certain class of sparsity-inducing cost penalty. In addition, we prove that under certain conditions on the solution, the LCA converges in a finite number of switches (i.e., node threshold crossings).
A. Charles, A.A. Kressner, and C.J.
Rozell.
Causal sparse decomposition of audio signals.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
Recent results have shown the utility of sparse coding for audio signals [citation]. While current inference methods can decompose audio signals, they require whole signals and are therefor ill suited for realtime applications that require causal processing. We propose a neurally inspired, causal, sparse inference scheme based on the Locally Competitive Algorithm (LCA) (Rozell et al. 2008) over a temporal-spectral neighborhood. We demonstrate that this causal inference scheme can achieve lower sparsity levels and better signal fidelity than current filter and threshold approaches. Additionally, for some regimes, the sparsity level approaches those of Matching Pursuit while still maintaining signal integrity.
H.L. Yap and C.J. Rozell.
Stable Takens' embedding for linear dynamical systems.
In Proceedings of the IEEE Conference on Decision and
Control, Atlanta, GA, December 2010.
Invited paper for session on Exploiting Sparsity and
Compressive Sensing in System Identification.
Takens' Embedding Theorem gives theoretical justification for the use of delay coordinate maps in characterizing and predicting nonlinear dynamical systems. However, in practice imperfections such as system and measurement noise may render these results unusable. In this paper, we consider conditions allowing for a stable version of Takens' Embedding Theorem in the restricted case of linear dynamical systems. Our work is inspired from results from the field of Compressive Sensing, where signals from a low-dimensional signal family residing in a high-dimensional space can be robustly recovered from compressive measurements only if the measurement form a stable embedding of the signal family. In particular, we show that a stable embedding of the attractor of the dynamical system is indeed possible
and give sufficient conditions on the number of delays and the observation function for the delay coordinate maps to be stabilized. In addition, we also show that when the attractor is an ellipse, the conditioning of the embedding is lower bounded by a positive constant dependent only on the dynamical system and not within control of the experimentalist. We illustrate our results with an example linear dynamical system converging to an elliptical attractor. Our analysis in this paper will give insights into stable Takens' Embedding of general dynamical systems.
C.J. Rozell and P. Garrigues.
Analog sparse approximation for compressed sensing recovery.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
Non-smooth convex optimization programs such as L1 minimization produce state-of-the-art results in many signal and image processing applications. Despite the progress in algorithms to solve these programs, they are still too computationally expensive for many real-time applications. Using recent results describing dynamical systems that efficiently solve these types of programs, we demonstrate through simulation that custom analog ICs implementations of this system could potentially perform compressed sensing recovery for real time applications approaching 500 KHz. Furthermore, we show that this architecture can implement several other optimization programs of recent interest, including reweighted L1 and group L1 minimization.
A. Charles and C.J. Rozell.
Sparse coding for spectral signatures in hyperspectral images.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
The growing use of hyperspectral imagery lead us to seek automated algorithms for extracting useful information about the scene. Recent work in sparse approximation has shown that unsupervised learning techniques can use example data to determine an efficient dictionary with few a priori assumptions. We apply this model to sample hyperspectral data and show that these techniques learn a dictionary that: 1) contains a meaningful spectral decomposition for hyperspectral imagery, 2) admit representations that are useful in determining properties and classifying materials in the scene, and 3) forms local approximations to the nonlinear manifold structure present in the actual data.
M.B. Wakin, J.Y. Park, H.L. Yap, and C.J. Rozell.
Concentration of measure for block diagonal measurement matrices.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Dallas, TX, March 2010.
[ .pdf ]
Concentration of measure inequalities are at the heart of much
theoretical analysis of randomized compressive operators. Though
commonly studied for dense matrices, in this paper we derive a
concentration of measure bound for block diagonal matrices where the
nonzero entries along the main diagonal blocks are i.i.d.
subgaussian random variables. Our main result states that the
concentration exponent, in the best case, scales as that for a fully
dense matrix. We also identify the role that the energy distribution
of the signal plays in distinguishing the best case from the worst.
We illustrate these phenomena with a series of experiments.
C.J. Rozell, H.L. Yap, J.Y. Park, and M.B. Wakin.
Concentration of measure for block diagonal matrices with repeated
blocks.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2010.
Invited paper.
[ .pdf ]
The theoretical analysis of randomized compressive operators often relies on the existence of a concentration of measure inequality for the operator of interest. Though commonly studied for unstructured, dense matrices, matrices with more structure are often of interest because they model constraints on the sensing system or allow more efficient system implementations. In this paper we derive a concentration of measure bound for block diagonal matrices where the nonzero entries along the main diagonal are a single repeated block of i.i.d. Gaussian random variables. Our main result states that the concentration exponent, in the best case, scales as that for a fully dense matrix. We also identify the role that the signal diversity plays in distinguishing the best and worst cases. Finally, we illustrate these phenomena with a series of experiments.
R.L. Ortman, C.J. Rozell, and D.H. Johnson.
Reconstruction of compressively sensed images via neurally plausible
local competitive algorithms.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), pages 476-479, Princeton, NJ, March 2008.
C.J. Rozell.
Distributed processing in frames for sparse approximation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2008.
Invited paper.
[ .pdf ]
Beyond signal processing applications, frames are also powerful tools
for modeling the sensing and information processing of many biological
and man-made systems that exhibit inherent redundancy. In many cases,
these systems are required to use distributed computational strategies
to analyze and process the sensory information. In this talk, I will
review the use of frames to model distributed sensing systems with a
particular focus on sensory neural systems. In light of the evidence
that many of these systems employ sparse codes, I will describe our
Locally Competitive Algorithms (LCAs) that use a dynamical system to
solve many sparse approximation problems. These LCAs employ a
parallel computational architecture with simple analog components. I
will show numerical simulation results for these systems and describe
their relationship to the many recently-proposed iterative
thresholding algorithms. Our LCA approach also demonstrates potential
advantages in coding time-varying signals (e.g., video) by reflecting
the smooth signal changes in smooth coefficient variations. Finally, I
will highlight some future directions where we hope to impact areas
such as efficient analog signal processing devices, fast discrete
approximation algorithms, and video processing and computer vision in
complex temporal environments.
C.J. Rozell, D.H. Johnson, R.G. Baraniuk, and B.A. Olshausen.
Locally competitive algorithms for sparse approximation.
In Proceedings of the International Conference on Image
Processing (ICIP), pages 169-172, San Antonio, TX, September 2007.
P. Casazza, G. Kutyniok, S. Li, and C.J. Rozell.
Modeling sensor networks with fusion frames.
In Proceedings of SPIE, Wavelets XII at SPIE Optics and
Photonics, volume 6701, pages 67011M-1 - 67011M-11, San Diego, CA, August
2007.
[ .pdf ]
The new notion of fusion frames will be presented in
this article. Fusion frames provide an extensive
framework not only to model sensor networks, but
also to serve as a means to improve robustness or
develop efficient and feasible reconstruction
algorithms. Fusion frames can be regarded as sets of
redundant subspaces each of which contains a
spanning set of local frame vectors, where the
subspaces have to satisfy special overlapping
properties. Main aspects of the theory of fusion
frames will be presented with a particular focus on
the design of sensor networks. New results on the
construction of Parseval fusion frames will also be
discussed.
C.J. Rozell and D.H. Johnson.
Power scheduling for wireless sensor and actuator networks.
In Proceedings of the International Conference on Information
Processing in Sensor Networks (IPSN), pages 470-478, Cambridge, MA, April
2007.
(Acceptance rate of 38/170˜22%.).
[ .pdf ]
We previously presented a model for some wireless sensor and actuator
network (WSAN) applications based on the vector space tools of frame
theory. In this WSAN model there is a weight associated to each
sensor-actuator link denoting the importance of that communication
link to the actuation fidelity. These weights were shown to be useful
in pruning away communication links to reduce the number of active
channels. Inspired by recent work in power scheduling for
decentralized estimation, we investigate the optimal allocation of
system resources for achieving a desired actuation fidelity. In this
scheme, each sensor acquires a noisy observation and sends a message
to a subset of actuators using an MQAM transmission strategy. The
message sent on each sensor-actuator communication link is quantized
with a variable number of bits, with the number of bits optimized to
minimize the total network power consumption subject to a constraint
on the actuation distortion. We show analytically and verify through
simulation that performing this optimal power scheduling can yield
significant power savings over communication strategies that use a
fixed number of bits on each communication link.
S.W. Bishnoi, C.S. Levin, C.J. Rozell, B.R. Johnson, D.H. Johnson, and N.J
Halas.
All-optical nanoscale pH meter: a plasmonic nanodevice with
quantifiable output.
In Proceedings of the Annual Meeting of the IEEE Lasers and
Electro-Optics Society (IEEE LEOS), Montreal, Canada, October 2006.
Invited paper.
C.J. Rozell and D.H. Johnson.
Evaluating local contributions to global performance in wireless
sensor and actuator networks.
Lecture Notes in Computer Science, 4026:1-16, June 2006.
Proceedings of the International Conference on Distributed
Computing in Sensor Systems (DCOSS), San Francisco, CA, June 2006.
[ .pdf ]
Wireless sensor networks are often studied with the goal of removing
information from the network as efficiently as possible. However,
when the application also includes an actuator network, it is
advantageous to determine actions in-network. In such settings,
optimizing the sensor node behavior with respect to sensor information
fidelity does not necessarily translate into optimum behavior in terms
of action fidelity. Inspired by neural systems, we present a model of
a sensor and actuator network based on the vector space tools of
frame theory that applies to applications analogous to reflex
behaviors in biological systems. Our analysis yields bounds on both
absolute and average actuation error that point directly to strategies
for limiting sensor communication based not only on local measurements
but also on a measure of how important each sensor-actuator link is to
the fidelity of the total actuation output.
C.J. Rozell, I.N. Goodman, and D.H. Johnson.
Feature-based information processing with selective attention.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Toulouse, France, May 2006.
[ .pdf ]
We present a simple but general model for feature-based
information processing with selective attention. We model
feature extraction as projections onto frames of subspaces,
which accounts for redundancies in the representations of
individual features as well as between features. To manage
limited resources, we use feedback attentional signals to
dynamically allocate system resources according to the
observed events. In our model, attention maximizes the
average information retained about all events weighted by
their relative priorities. We illustrate the model with a
simple system under a total bit constraint and discuss how
the organization of the feature extraction affects the
optimal bit allocation.
C.J. Rozell and D.H. Johnson.
Analysis of noise reduction in redundant expansions under distributed
processing requirements.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 185-188, Philadelphia, PA,
March 2005.
[ .pdf ]
We considered signal reconstruction with
redundant expansions under distributed processing in noisy environments.
Redundant expansions have the ability to reduce noise corrupting the
coefficients, but distributed processing schemes will not be able
to take full advantage of the redundancy present. We apply frame
theory and a generalization called “frames of subspaces” to find
conditions when distributed reconstruction suffers no loss in noise
reduction ability, and we bound performance loss in more general cases.
M.A. Lexa, C.J. Rozell, S. Sinanović, and D.H. Johnson.
To cooperate or not to cooperate: Detection strategies in sensor
networks.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 841-844, Montreal, Canada,
May 2004.
[ .pdf ]
This paper is an initial investigation into the following
question: Can cooperation among sensors in a sensor network improve
detection performance in a simple hypothesis test? We analyze a simple
cooperative system using the Kullback-Leibler (KL) discrimination distance
and a quantity known as the information transfer ratio which is a
ratio of KL distances. We discover that, asymptotically, gain over a
non-cooperative system depends on the conditional KL distance. We conclude
with an illustrative example which demonstrates that cooperation not
only significantly improves performance but can also degrade it.
C.J. Rozell and D. Manolakis.
Matched filter performance for unequal target and background
covariance matrices.
In Proceedings of the SPIE Defense and Security Symposium:
Algorithms and Technologies for Multispectral, Hyperspectral, and
Ultraspectral Imagery X, pages 109-117, Orlando, FL, April 2004.
Detection of military and civilian targets from
airborne platforms using hyperspectral imaging (HSI) sensors is of
great interest. Relative to multispectral sensing, hyperspectral
sensing can increase the detectability of targets by exploiting finer
detail in spectral signatures. A multitude of adaptive detection
algorithms have appeared in the literature or have found their way into
software packages and end-user systems. The most widely known among them
is the linear matched filter. However, despite its popularity, the
fact that the matched filter is used under conditions that deviate
from the implicit optimality assumptions has not been investigated.
M. Simoni, B. Broening, C. Rozell, C. Meek, and G. Wakefield.
A theoretical framework for electro-acoustic music.
In International Computer Music Conference (ICMC), Beijing,
China, 1999.
[ .pdf ]
A. Balavoine, J. Romberg, and C.J. Rozell.
Convergence and rate analysis of neural networks for sparse
approximation.
2012.
In review. Submitted July 2011. Revision posted March 27, 2012.
[ .pdf ]
We present an analysis of the Locally Competitive Algorithm (LCA), a Hopfield-style neural network that efficiently solves sparse approximation problems (e.g., approximating a vector from a dictionary using just a few non-zero coefficients). This class of problems plays a significant role in both theories of neural coding and applications in signal processing. However, the LCA lacks analysis of its convergence properties and previous results on neural networks for nonsmooth optimization do not apply to the specifics of the LCA architecture. We show that the LCA has desirable convergence properties, such as stability and global convergence to the optimum of the objective function when it is unique. Under some mild conditions, the support of the solution is also proven to be reached in finite time. Furthermore, some restrictions on the problem specifics allow us to characterize the convergence rate of the system by showing that the LCA converges exponentially fast with an analytically bounded convergence rate. We support our analysis with several illustrative simulations.
S. Shapero, C. Rozell, and P. Hasler.
Low power sparse approximation on reconfigurable analog hardware.
2012.
In review. Submitted November 2011.
Compressed sensing is an important optimization problem in signal and image processing applications. A Hopfield-Network-like analog system is proposed as a solution, using the Locally Competitive Algorithm (LCA) to solve an overcomplete l1 sparse approximation problem. A scalable system architecture using sub-threshold currents is described, including vector matrix multipliers (VMMs) and a nonlinear thresholder. A 4x6 nonlinear system is implemented on the RASP 2.9v chip, a Field Programmable Analog Array with directly programmable floating gate elements, allowing highly accurate VMMs. The circuit successfully reproduced the outputs of a digital optimization program, converging to within 4.8% RMS, and an optimization cost only 1.3% higher on average. The active circuit consumed 29μA of current at 2.4V, and converges on solutions in 240μs. A smaller 2x3 system is also implemented. Extrapolating the scaling trends to N=1000 node system, the Analog LCA compares favorably with State-of-the-Art digital solutions.
M. Zhu and C.J. Rozell.
Visual nonclassical receptive field effects emerge from sparse coding
in a dynamical system.
2012.
In review. Submitted April 2012.
Extensive electrophysiology studies have shown that many V1 simple cells receive contextual influence, where stimuli not part of the classical receptive field (CRF) can modulate the cell’s response to CRF stimuli. Mechanistic models of these non-classical receptive field (nCRF) effects do not address the role of these effects in the optimal coding or processing of natural stimuli. We demonstrate that a wide variety of nCRF effects are emergent properties of a single sparse coding model implemented in a neurally plausible network structure (requiring no tuning of any parameters to produce the various effects). In particular, this model produces both individual cells showing canonical nCRF effects, as well as the population diversity reported in the physiology literature. These results show that the sparse coding hypothesis, when coupled with a biophysically plausible implementation, can provide apparently disparate response properties with a unified interpretation based on the efficient coding of natural scenes.
A.A. Kressner, D.V. Anderson, and C.J. Rozell.
Evaluating the generalization of the hearing aid speech quality index
(HASQI).
2012.
In review. Submitted April 2012.
Many developers of audio signal processing strategies rely on objective measures of quality for initial evaluations of algorithms. As such, objective measures should be robust, and they should be able to predict quality accurately regardless of the dataset or testing conditions. Kates and Arehart have developed the Hearing Aid Speech Quality Index (HASQI) to predict the effects of noise, nonlinear distortion, and linear filtering on speech quality for both normal-hearing and hearing-impaired listeners, and they report very high performance with their training and testing datasets [Kates, J. and Arehart, K., Audio Eng. Soc., 58(5), 363-381 (2010)]. In order to investigate the generalizability of HASQI, we test its ability to predict normal-hearing listeners’ subjective quality ratings of a dataset on which it was not trained. This dataset is designed specifically to contain a wide range of distortions introduced by real-world noises which have been processed by some of the most common noise suppression algorithms in hearing aids. We show that HASQI achieves prediction performance comparable to PESQ, the standard for objective measures of quality, as well as some of the other measures in the literature. Furthermore, we identify areas of weakness and show that training can improve quantitative prediction.
A.S. Charles, P. Garrigues, and C.J. Rozell.
A common network architecture can efficiently implement diverse
sparsity-based inference problems.
2012.
In review. Submitted May 2012.
Recent signal processing developments have greatly expanded the range of sparsity based optimization functions for a variety of statistical models. Initial work in finding neurally plausible systems resulted in the locally competitive algorithms for solving basic sparsity inducing optimizations. Incorporating these new model types, however, would greatly increase the relevance of neural systems for solving regularized least-squares problems. In this work we show analytically that a wide range of sparse approximation problems can be solved in the LCA architecture, including approximate lp norms, modified l1 norms, re-weighted l1 and l2, the block l1 norm and classic Tikhonov regularization.
J.Y. Park, H.L. Yap, C.J. Rozell, and M.B. Wakin.
Concentration of measure for block diagonal matrices with
applications to compressive signal processing.
IEEE Transactions on Signal Processing, 59:5859-5875, December
2011.
[ .pdf ]
Theoretical analysis of randomized, compressive operators often depends on a concentration of measure inequality for the operator in question. Typically, such inequalities quantify the likelihood that a random matrix will preserve the norm of a signal after multiplication. Concentration of measure results are well-established for unstructured compressive matrices, populated with independent and identically distributed (i.i.d.) random entries. Many real-world acquisition systems, however, are subject to architectural constraints that make such matrices impractical. In this paper we derive concentration of measure bounds for two types of block diagonal compressive matrices, one in which the blocks along the main diagonal are random and independent, and one in which the blocks are random but equal. For both types of matrices, we show that the likelihood of norm preservation depends on certain properties of the signal being measured, but that for the best case signals, both types of block diagonal matrices can offer concentration performance on par with their unstructured, i.i.d. counterparts. We support our theoretical results with illustrative simulations as well as analytical and empirical investigations of several signal classes that are highly amenable to measurement using block diagonal matrices. We also discuss applications of these results in ensuring stable embeddings for various signal families and in establishing performance guarantees for
solving various signal processing tasks (such as detection and classification) directly in the compressed domain.
H.L. Yap and C.J. Rozell.
Stable Takens' embeddings for linear dynamical systems.
IEEE Transactions on Signal Processing, 59(10):4781-4794,
October 2011.
[ .pdf ]
Takens' Embedding Theorem remarkably established that concatenating M previous outputs of a dynamical system into a vector (called a delay coordinate map) can be a one-to-one mapping of a low-dimensional attractor from the system state-space.
However, Takens' theorem is fragile because even small imperfections can induce arbitrarily large errors in the attractor representation. We extend Takens' result to establish explicit, non-asymptotic sufficient conditions for a delay coordinate map to form a stable embedding in the restricted case of linear dynamical systems and observation functions.
Our work is inspired by the field of Compressive Sensing (CS), where results guarantee that low-dimensional signal families can be robustly reconstructed if they are stably embedded by a measurement operator.
However, in contrast to typical CS results, i) our sufficient conditions are independent of the size of the ambient state space (N), and ii) some system and measurement pairs have fundamental limits on the conditioning of the embedding (i.e., how close it is to an isometry), meaning that further measurements beyond some point add no further significant value.
We use several simple simulations to explore the conditions of the main results, including the tightness of the bounds and the convergence speed of the stable embedding.
A.S. Charles, B.A. Olshausen, and C.J. Rozell.
Learning sparse codes for hyperspectral imagery.
IEEE Journal of Selected Topics in Signal Processing,
5(5):963-978, September 2011.
[ .pdf ]
The spectral features in hyperspectral imagery (HSI) contain significant structure that, if properly characterized could enable more efficient data acquisition and improved data analysis. Because most pixels contain reflectances of just a few materials, we propose that a sparse coding model is well-matched to HSI data. Sparsity models consider each pixel as a combination of just a few elements from a larger dictionary, and this approach has proven effective in a wide range of applications. Furthermore, previous work has shown that optimal sparse coding dictionaries can be learned from a dataset with no other a priori information (in contrast to many HSI “endmember” discovery algorithms that assume the presence of pure spectra or side information).
We modified an existing unsupervised learning approach and applied it to HSI data (with significant ground truth labeling) to learn an optimal sparse coding dictionary. Using this learned dictionary, we demonstrate three main findings: i) the sparse coding model learns spectral signatures of materials in the scene and locally approximates nonlinear manifolds for individual materials, ii) this learned dictionary can be used to infer HSI-resolution data with very high accuracy from simulated imagery collected at multispectral-level resolution, and iii) this learned dictionary improves the performance of a supervised classification algorithm, both in terms of the classifier complexity and generalization from very small training sets.
C.J. Rozell, D.H Johnson, R.G. Baraniuk, and B.A. Olshausen.
Sparse coding via thresholding and local competition in neural
circuits.
Neural Computation, 20(10):2526-2563, October 2008.
[ .pdf ]
While evidence indicates that neural systems may be
employing sparse approximations to represent sensed
stimuli, the mechanisms underlying this ability are
not understood. We describe a local ly competitive
algorithm (LCA) that solves a collection of sparse
coding principles minimizing a weighted combination
of mean-squared error (MSE) and a coefficient cost
function. LCAs are designed to be implemented in a
dynamical system composed of many neuron-like
elements operating in parallel. These algorithms use
thresholding functions to induce local (usually
one-way) inhibitory competitions between nodes to
produce sparse representations. LCAs produce
coefficients with sparsity levels comparable to the
most popular centralized sparse coding algorithms
while being readily suited for neural
implementation. Addi- tionally, LCA coefficients for
video sequences demonstrate inertial properties that
are both qualitatively and quantitatively more
regular (i.e., smoother and more predictable) than
the coefficients produced by greedy algorithms.
We show that an Au nanoshell with a pH sensitive
molecular adsorbate functions as a standalone, all-optical nanoscale
pH meter that monitors its local environment through the
pH-dependent surface enhanced Raman scattering (SERS) spectra of the
adsorbate molecules. Moreover, we also show how the performance of
such a functional nanodevice can be quantitatively assessed. The
complex spectral output is reduced to a simple device characteristic
by application of a locally linear manifold approximation
algorithm. The average accuracy of the nano-“meter” was found to
be ± 0.10 pH units across its operating range.
C.J. Rozell and D.H. Johnson.
Analyzing the robustness of redundant population codes in sensory and
feature extraction systems.
Neurocomputing, 69(10-12):1215-1218, June 2006.
[ .pdf ]
Sensory systems often use groups of redundant neurons to represent
stimulus information both during transduction and population coding of
features. This redundancy makes the system more robust to corruption
in the representation. We approximate neural coding as a projection of
the stimulus onto a set of vectors, with the result encoded by spike
trains. We use the formalism of frame theory to quantify the inherent
noise reduction properties of such population codes. Additionally,
computing features from the stimulus signal can also be thought of as
projecting the coefficients of a sensory representation onto another
set of vectors specific to the feature of interest. The conditions
under which a combination of different features form a complete
representation for the stimulus signal can be found through a recent
extension to frame theory called “frames of subspaces.” We extend
the frame of subspaces theory to quantify the noise reduction
properties of a collection of redundant feature spaces.
C.J. Rozell and D.H. Johnson.
Examining methods for estimating mutual information in spiking neural
systems.
Neurocomputing, 65-66C:429-434, June 2005.
[ .pdf ]
Mutual information enjoys wide use in the computational neuroscience community for
analyzing spiking neural systems. Its direct calculation is difficult
because estimating the joint stimulus-response distribution requires a
prohibitive amount of data. Consequently, several techniques have
appeared for bounding mutual information that rely on less data. We
examine two upper bound techniques and find that they are either
unreliable or introduce strong assumptions about the neural code. We also
examine two lower bounds, showing that they can be very loose and
possibly bear little relation to the mutual information's actual value.
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Measuring information transfer in crayfish sustaining fiber spike
generators.
Biological Cybernetics, 90(2):89-97, February 2004.
[ .pdf ]
We present a method based
on information-theoretic distances for measuring the information
transfer efficiency of voltage to impulse encoders. In response to light
pulses, we simultaneously recorded the EPSP and spiking output of
crayfish sustaining fibers. To measure the distance between analog
EPSP responses, we developed a membrane noise model that accurately
captures stimulus-induced nonstationarities. By comparing the EPSP
and spike responses, we found encoding efficiencies on the order
of 10-4, with interesting dynamics occurring during initial
transients. A simple analog to point-process converter predicted the small
information transfer efficiencies and dynamic properties we measured.
Copyright held by Springer-Verlag. The original
publication is available at springerlink.com. DOI:10.1007/s00422-003-0458-y
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Information processing during transient responses in the crayfish
visual system.
Neurocomputing, 52-54:53-58, June 2003.
[ .pdf ]
We analyzed sustaining fiber
responses in the crayfish visual system to light pulses using information
processing techniques. The light pulse stimuli elicited a transient and
a steady-state component in the EPSP input and in the firing rate
of the spike train output. The overall information transfer of the
system was very low (10-4), with a sharp increase during the
transient portion of the response followed by a steady decrease. The
information transfer dynamics are consistent with a simple spike
generator model that depends explicitly on stimulus changes. The
present analysis also corroborates the observed light reflex behavior.
D.H. Johnson, I.N. Goodman, and C.J. Rozell.
Information theory and systems neuroscience.
In S. Grün and S. Rotter, editors, Analysis of parallel
spike trains. Springer-Verlag, 2010.
H.L. Yap, A. Charles, and C.J.
Rozell.
The restricted isometry property for echo state networks with
applications to sequence memory capacity.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
D. Sale, C.J. Rozell, J.K. Romberg, and A.D. Lanterman.
Compressive LADAR in realistic environments.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
S. Shapero, C. Rozell, A. Balavoine, and P. Hasler.
A scalable implementation of sparse approximation on a Field
Programmable Analog Array.
In IEEE Biomedical Circuits and Systems Conference (BioCAS), La
Jolla, CA, November 2011.
A. Kressner, D. Anderson, and C. Rozell.
Robustness of the hearing aid speech quality index (HASQI).
In IEEE Workshop on Applications of Signal Processing to Audio
and Acoustics (WASPAA), New Paltz, NY, October 2011.
[ .pdf ]
Objective measures of speech quality have been the subject of significant prior work, particularly in the areas of speech codecs and communication channels for normal-hearing listeners. One of the primary concerns of researchers in this area is how these metrics generalize to datasets or listener studies which are “unknown” to the measures. Another growing concern is how these metrics perform for the hearing-impaired community. Researchers working with the this community need to be able to predict how hearing-impaired listeners will perceive the quality of speech, as well as how they will perceive the quality of speech processed specifically by hear- ing aids. A relatively recent metric, the Hearing Aid Speech Quality Index (HASQI), is a model-based objective measure of quality developed in the context of hearing aids for normal-hearing and hearing-impaired listeners (Kates & Arehart, Journal of the Audio Engineering Society, 2010). As such, HASQI makes substantial progress on some of the generalization issues. However, HASQI has not been tested thus far on any datasets other than the one on which it was trained. The objective of this study is to demonstrate the robustness of HASQI in predicting subjective quality. We use an “unknown” dataset of noisy speech processed by noise suppression algorithms, along with a corresponding set of subjective quality scores from normal-hearing listeners, to demonstrate HASQI’s prediction performance. Furthermore, we compare HASQI’s performance with that of several other objective measures in order to provide a point of reference.
M.S. Asif, A. Charles, J. Romberg, and C. Rozell.
Estimation and dynamic updating of time-varying signals with sparse
variations.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Prague, Czech Republic, May 2011.
[ .pdf ]
This paper presents an algorithm for an l1-regularized Kalman filter. Given observations of a discrete-time linear dynamical system with sparse errors in the state evolution, we estimate the state sequence by solving an optimization algorithm that balances fidelity to the measurements (measured by the standard l2 norm) against the sparsity of the innovations (measured using the l1 norm). We also derive an efficient algorithm for updating the estimate as the system evolves. This dynamic updating algorithm uses a homotopy scheme that tracks the solution as new measurements are slowly worked into the system and old measurements are slowly removed. The effective cost of adding new measurements is a number of low-rank updates to the solution of a linear system of equations that is roughly proportional to the joint sparsity of all the innovations in the time interval of interest.
H.L. Yap, M.B. Wakin, and C.J. Rozell.
Stable manifold embeddings with operators satisfying the restricted
isometry property.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
Signals of interests can often be thought to come from a low dimensional signal model.
The exploitation of this fact has led to many recent interesting advances in signal processing, one notable example being in the field of compressive sensing (CS).
The literature on CS has established that many matrices satisfy the Restricted Isometry Property (RIP), which guarantees a stable (i.e., distance-preserving) embedding of a sparse signal model from an undersampled linear measurement system.
In this work, we study the stable embedding of manifold signal models using matrices that satisfy the RIP.
We show that by paying reasonable additional factors in the number of measurements, all matrices that satisfy the RIP can also be used (in conjunction with a random sign sequence) to obtain a stable embedding of a manifold.
H.L. Yap, A. Eftekhari, M.B. Wakin, and C.J. Rozell.
The restricted isometry property for block diagonal matrices.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In compressive sensing (CS), the Restricted Isometry Property (RIP) is a powerful condition on measurement operators which ensures robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms.
Early papers in CS showed that Gaussian random matrices satisfy the RIP with high probability, but such matrices are usually undesirable in practical applications due to storage limitations, computational considerations, or the mismatch of such matrices with certain measurement architectures.
To alleviate some or all of these difficulties, recent research efforts have focused on structured random matrices.
In this paper, we study block diagonal measurement matrices where each block on the main diagonal is itself a Gaussian random matrix.
The main result of this paper shows that such matrices can indeed satisfy the RIP but that the requisite number of measurements depends on the coherence of the basis in which the signals are sparse.
In the best case-for signals that are sparse in the frequency domain-these matrices perform nearly as well as dense Gaussian random matrices despite having many fewer nonzero entries.
A. Charles, M.S. Asif, J. Romberg, and C. Rozell.
Sparsity penalties in dynamical system estimation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In this work we address the problem of state estimation in dynamical systems using recent developments in compressive sensing and sparse approximation. We formulate the traditional Kalman filter as a one-step update optimization procedure which leads us to a more unified framework, useful for incorporating sparsity constraints. We introduce three combinations of two sparsity conditions (sparsity in the state and sparsity in the innovations) and write recursive optimization programs to estimate the state for each model. This paper is meant as an overview of different methods for incorporating sparsity into the dynamic model, a presentation of algorithms that unify the support and coefficient estimation, and a demonstration that these suboptimal schemes can actually show some performance improvements (either in estimation error or convergence time) over standard optimal methods that use an impoverished model.
A. Balavoine, C.J. Rozell, and J.K. Romberg.
Global convergence of the Locally Competitive Algorithm.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
The Locally Competitive Algorithm (LCA) is a continuous-time dynamical system designed to solve the problem of sparse approximation. This class of approximation problems plays an important role in producing state-of-the-art results in many signal processing and inverse problems, and implementing the LCA in analog VLSI may significantly improve the time and power necessary to solve these optimization programs. The goal of this paper is to analyze the dynamical behavior of the LCA system and guarantee its convergence and stability. We show that the LCA converges from any initial point. We also show that fixed points of the system are extrema of the sparse approximation objective function when designed for a certain class of sparsity-inducing cost penalty. In addition, we prove that under certain conditions on the solution, the LCA converges in a finite number of switches (i.e., node threshold crossings).
A. Charles, A.A. Kressner, and C.J.
Rozell.
Causal sparse decomposition of audio signals.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
Recent results have shown the utility of sparse coding for audio signals [citation]. While current inference methods can decompose audio signals, they require whole signals and are therefor ill suited for realtime applications that require causal processing. We propose a neurally inspired, causal, sparse inference scheme based on the Locally Competitive Algorithm (LCA) (Rozell et al. 2008) over a temporal-spectral neighborhood. We demonstrate that this causal inference scheme can achieve lower sparsity levels and better signal fidelity than current filter and threshold approaches. Additionally, for some regimes, the sparsity level approaches those of Matching Pursuit while still maintaining signal integrity.
H.L. Yap and C.J. Rozell.
Stable Takens' embedding for linear dynamical systems.
In Proceedings of the IEEE Conference on Decision and
Control, Atlanta, GA, December 2010.
Invited paper for session on Exploiting Sparsity and
Compressive Sensing in System Identification.
Takens' Embedding Theorem gives theoretical justification for the use of delay coordinate maps in characterizing and predicting nonlinear dynamical systems. However, in practice imperfections such as system and measurement noise may render these results unusable. In this paper, we consider conditions allowing for a stable version of Takens' Embedding Theorem in the restricted case of linear dynamical systems. Our work is inspired from results from the field of Compressive Sensing, where signals from a low-dimensional signal family residing in a high-dimensional space can be robustly recovered from compressive measurements only if the measurement form a stable embedding of the signal family. In particular, we show that a stable embedding of the attractor of the dynamical system is indeed possible
and give sufficient conditions on the number of delays and the observation function for the delay coordinate maps to be stabilized. In addition, we also show that when the attractor is an ellipse, the conditioning of the embedding is lower bounded by a positive constant dependent only on the dynamical system and not within control of the experimentalist. We illustrate our results with an example linear dynamical system converging to an elliptical attractor. Our analysis in this paper will give insights into stable Takens' Embedding of general dynamical systems.
C.J. Rozell and P. Garrigues.
Analog sparse approximation for compressed sensing recovery.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
Non-smooth convex optimization programs such as L1 minimization produce state-of-the-art results in many signal and image processing applications. Despite the progress in algorithms to solve these programs, they are still too computationally expensive for many real-time applications. Using recent results describing dynamical systems that efficiently solve these types of programs, we demonstrate through simulation that custom analog ICs implementations of this system could potentially perform compressed sensing recovery for real time applications approaching 500 KHz. Furthermore, we show that this architecture can implement several other optimization programs of recent interest, including reweighted L1 and group L1 minimization.
A. Charles and C.J. Rozell.
Sparse coding for spectral signatures in hyperspectral images.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
The growing use of hyperspectral imagery lead us to seek automated algorithms for extracting useful information about the scene. Recent work in sparse approximation has shown that unsupervised learning techniques can use example data to determine an efficient dictionary with few a priori assumptions. We apply this model to sample hyperspectral data and show that these techniques learn a dictionary that: 1) contains a meaningful spectral decomposition for hyperspectral imagery, 2) admit representations that are useful in determining properties and classifying materials in the scene, and 3) forms local approximations to the nonlinear manifold structure present in the actual data.
M.B. Wakin, J.Y. Park, H.L. Yap, and C.J. Rozell.
Concentration of measure for block diagonal measurement matrices.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Dallas, TX, March 2010.
[ .pdf ]
Concentration of measure inequalities are at the heart of much
theoretical analysis of randomized compressive operators. Though
commonly studied for dense matrices, in this paper we derive a
concentration of measure bound for block diagonal matrices where the
nonzero entries along the main diagonal blocks are i.i.d.
subgaussian random variables. Our main result states that the
concentration exponent, in the best case, scales as that for a fully
dense matrix. We also identify the role that the energy distribution
of the signal plays in distinguishing the best case from the worst.
We illustrate these phenomena with a series of experiments.
C.J. Rozell, H.L. Yap, J.Y. Park, and M.B. Wakin.
Concentration of measure for block diagonal matrices with repeated
blocks.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2010.
Invited paper.
[ .pdf ]
The theoretical analysis of randomized compressive operators often relies on the existence of a concentration of measure inequality for the operator of interest. Though commonly studied for unstructured, dense matrices, matrices with more structure are often of interest because they model constraints on the sensing system or allow more efficient system implementations. In this paper we derive a concentration of measure bound for block diagonal matrices where the nonzero entries along the main diagonal are a single repeated block of i.i.d. Gaussian random variables. Our main result states that the concentration exponent, in the best case, scales as that for a fully dense matrix. We also identify the role that the signal diversity plays in distinguishing the best and worst cases. Finally, we illustrate these phenomena with a series of experiments.
R.L. Ortman, C.J. Rozell, and D.H. Johnson.
Reconstruction of compressively sensed images via neurally plausible
local competitive algorithms.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), pages 476-479, Princeton, NJ, March 2008.
C.J. Rozell.
Distributed processing in frames for sparse approximation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2008.
Invited paper.
[ .pdf ]
Beyond signal processing applications, frames are also powerful tools
for modeling the sensing and information processing of many biological
and man-made systems that exhibit inherent redundancy. In many cases,
these systems are required to use distributed computational strategies
to analyze and process the sensory information. In this talk, I will
review the use of frames to model distributed sensing systems with a
particular focus on sensory neural systems. In light of the evidence
that many of these systems employ sparse codes, I will describe our
Locally Competitive Algorithms (LCAs) that use a dynamical system to
solve many sparse approximation problems. These LCAs employ a
parallel computational architecture with simple analog components. I
will show numerical simulation results for these systems and describe
their relationship to the many recently-proposed iterative
thresholding algorithms. Our LCA approach also demonstrates potential
advantages in coding time-varying signals (e.g., video) by reflecting
the smooth signal changes in smooth coefficient variations. Finally, I
will highlight some future directions where we hope to impact areas
such as efficient analog signal processing devices, fast discrete
approximation algorithms, and video processing and computer vision in
complex temporal environments.
C.J. Rozell, D.H. Johnson, R.G. Baraniuk, and B.A. Olshausen.
Locally competitive algorithms for sparse approximation.
In Proceedings of the International Conference on Image
Processing (ICIP), pages 169-172, San Antonio, TX, September 2007.
P. Casazza, G. Kutyniok, S. Li, and C.J. Rozell.
Modeling sensor networks with fusion frames.
In Proceedings of SPIE, Wavelets XII at SPIE Optics and
Photonics, volume 6701, pages 67011M-1 - 67011M-11, San Diego, CA, August
2007.
[ .pdf ]
The new notion of fusion frames will be presented in
this article. Fusion frames provide an extensive
framework not only to model sensor networks, but
also to serve as a means to improve robustness or
develop efficient and feasible reconstruction
algorithms. Fusion frames can be regarded as sets of
redundant subspaces each of which contains a
spanning set of local frame vectors, where the
subspaces have to satisfy special overlapping
properties. Main aspects of the theory of fusion
frames will be presented with a particular focus on
the design of sensor networks. New results on the
construction of Parseval fusion frames will also be
discussed.
C.J. Rozell and D.H. Johnson.
Power scheduling for wireless sensor and actuator networks.
In Proceedings of the International Conference on Information
Processing in Sensor Networks (IPSN), pages 470-478, Cambridge, MA, April
2007.
(Acceptance rate of 38/170˜22%.).
[ .pdf ]
We previously presented a model for some wireless sensor and actuator
network (WSAN) applications based on the vector space tools of frame
theory. In this WSAN model there is a weight associated to each
sensor-actuator link denoting the importance of that communication
link to the actuation fidelity. These weights were shown to be useful
in pruning away communication links to reduce the number of active
channels. Inspired by recent work in power scheduling for
decentralized estimation, we investigate the optimal allocation of
system resources for achieving a desired actuation fidelity. In this
scheme, each sensor acquires a noisy observation and sends a message
to a subset of actuators using an MQAM transmission strategy. The
message sent on each sensor-actuator communication link is quantized
with a variable number of bits, with the number of bits optimized to
minimize the total network power consumption subject to a constraint
on the actuation distortion. We show analytically and verify through
simulation that performing this optimal power scheduling can yield
significant power savings over communication strategies that use a
fixed number of bits on each communication link.
S.W. Bishnoi, C.S. Levin, C.J. Rozell, B.R. Johnson, D.H. Johnson, and N.J
Halas.
All-optical nanoscale pH meter: a plasmonic nanodevice with
quantifiable output.
In Proceedings of the Annual Meeting of the IEEE Lasers and
Electro-Optics Society (IEEE LEOS), Montreal, Canada, October 2006.
Invited paper.
C.J. Rozell and D.H. Johnson.
Evaluating local contributions to global performance in wireless
sensor and actuator networks.
Lecture Notes in Computer Science, 4026:1-16, June 2006.
Proceedings of the International Conference on Distributed
Computing in Sensor Systems (DCOSS), San Francisco, CA, June 2006.
[ .pdf ]
Wireless sensor networks are often studied with the goal of removing
information from the network as efficiently as possible. However,
when the application also includes an actuator network, it is
advantageous to determine actions in-network. In such settings,
optimizing the sensor node behavior with respect to sensor information
fidelity does not necessarily translate into optimum behavior in terms
of action fidelity. Inspired by neural systems, we present a model of
a sensor and actuator network based on the vector space tools of
frame theory that applies to applications analogous to reflex
behaviors in biological systems. Our analysis yields bounds on both
absolute and average actuation error that point directly to strategies
for limiting sensor communication based not only on local measurements
but also on a measure of how important each sensor-actuator link is to
the fidelity of the total actuation output.
C.J. Rozell, I.N. Goodman, and D.H. Johnson.
Feature-based information processing with selective attention.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Toulouse, France, May 2006.
[ .pdf ]
We present a simple but general model for feature-based
information processing with selective attention. We model
feature extraction as projections onto frames of subspaces,
which accounts for redundancies in the representations of
individual features as well as between features. To manage
limited resources, we use feedback attentional signals to
dynamically allocate system resources according to the
observed events. In our model, attention maximizes the
average information retained about all events weighted by
their relative priorities. We illustrate the model with a
simple system under a total bit constraint and discuss how
the organization of the feature extraction affects the
optimal bit allocation.
C.J. Rozell and D.H. Johnson.
Analysis of noise reduction in redundant expansions under distributed
processing requirements.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 185-188, Philadelphia, PA,
March 2005.
[ .pdf ]
We considered signal reconstruction with
redundant expansions under distributed processing in noisy environments.
Redundant expansions have the ability to reduce noise corrupting the
coefficients, but distributed processing schemes will not be able
to take full advantage of the redundancy present. We apply frame
theory and a generalization called “frames of subspaces” to find
conditions when distributed reconstruction suffers no loss in noise
reduction ability, and we bound performance loss in more general cases.
M.A. Lexa, C.J. Rozell, S. Sinanović, and D.H. Johnson.
To cooperate or not to cooperate: Detection strategies in sensor
networks.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 841-844, Montreal, Canada,
May 2004.
[ .pdf ]
This paper is an initial investigation into the following
question: Can cooperation among sensors in a sensor network improve
detection performance in a simple hypothesis test? We analyze a simple
cooperative system using the Kullback-Leibler (KL) discrimination distance
and a quantity known as the information transfer ratio which is a
ratio of KL distances. We discover that, asymptotically, gain over a
non-cooperative system depends on the conditional KL distance. We conclude
with an illustrative example which demonstrates that cooperation not
only significantly improves performance but can also degrade it.
C.J. Rozell and D. Manolakis.
Matched filter performance for unequal target and background
covariance matrices.
In Proceedings of the SPIE Defense and Security Symposium:
Algorithms and Technologies for Multispectral, Hyperspectral, and
Ultraspectral Imagery X, pages 109-117, Orlando, FL, April 2004.
Detection of military and civilian targets from
airborne platforms using hyperspectral imaging (HSI) sensors is of
great interest. Relative to multispectral sensing, hyperspectral
sensing can increase the detectability of targets by exploiting finer
detail in spectral signatures. A multitude of adaptive detection
algorithms have appeared in the literature or have found their way into
software packages and end-user systems. The most widely known among them
is the linear matched filter. However, despite its popularity, the
fact that the matched filter is used under conditions that deviate
from the implicit optimality assumptions has not been investigated.
M. Simoni, B. Broening, C. Rozell, C. Meek, and G. Wakefield.
A theoretical framework for electro-acoustic music.
In International Computer Music Conference (ICMC), Beijing,
China, 1999.
[ .pdf ]
A. Balavoine, J. Romberg, and C.J. Rozell.
Convergence and rate analysis of neural networks for sparse
approximation.
2012.
In review. Submitted July 2011. Revision posted March 27, 2012.
[ .pdf ]
We present an analysis of the Locally Competitive Algorithm (LCA), a Hopfield-style neural network that efficiently solves sparse approximation problems (e.g., approximating a vector from a dictionary using just a few non-zero coefficients). This class of problems plays a significant role in both theories of neural coding and applications in signal processing. However, the LCA lacks analysis of its convergence properties and previous results on neural networks for nonsmooth optimization do not apply to the specifics of the LCA architecture. We show that the LCA has desirable convergence properties, such as stability and global convergence to the optimum of the objective function when it is unique. Under some mild conditions, the support of the solution is also proven to be reached in finite time. Furthermore, some restrictions on the problem specifics allow us to characterize the convergence rate of the system by showing that the LCA converges exponentially fast with an analytically bounded convergence rate. We support our analysis with several illustrative simulations.
S. Shapero, C. Rozell, and P. Hasler.
Low power sparse approximation on reconfigurable analog hardware.
2012.
In review. Submitted November 2011.
Compressed sensing is an important optimization problem in signal and image processing applications. A Hopfield-Network-like analog system is proposed as a solution, using the Locally Competitive Algorithm (LCA) to solve an overcomplete l1 sparse approximation problem. A scalable system architecture using sub-threshold currents is described, including vector matrix multipliers (VMMs) and a nonlinear thresholder. A 4x6 nonlinear system is implemented on the RASP 2.9v chip, a Field Programmable Analog Array with directly programmable floating gate elements, allowing highly accurate VMMs. The circuit successfully reproduced the outputs of a digital optimization program, converging to within 4.8% RMS, and an optimization cost only 1.3% higher on average. The active circuit consumed 29μA of current at 2.4V, and converges on solutions in 240μs. A smaller 2x3 system is also implemented. Extrapolating the scaling trends to N=1000 node system, the Analog LCA compares favorably with State-of-the-Art digital solutions.
M. Zhu and C.J. Rozell.
Visual nonclassical receptive field effects emerge from sparse coding
in a dynamical system.
2012.
In review. Submitted April 2012.
Extensive electrophysiology studies have shown that many V1 simple cells receive contextual influence, where stimuli not part of the classical receptive field (CRF) can modulate the cell’s response to CRF stimuli. Mechanistic models of these non-classical receptive field (nCRF) effects do not address the role of these effects in the optimal coding or processing of natural stimuli. We demonstrate that a wide variety of nCRF effects are emergent properties of a single sparse coding model implemented in a neurally plausible network structure (requiring no tuning of any parameters to produce the various effects). In particular, this model produces both individual cells showing canonical nCRF effects, as well as the population diversity reported in the physiology literature. These results show that the sparse coding hypothesis, when coupled with a biophysically plausible implementation, can provide apparently disparate response properties with a unified interpretation based on the efficient coding of natural scenes.
A.A. Kressner, D.V. Anderson, and C.J. Rozell.
Evaluating the generalization of the hearing aid speech quality index
(HASQI).
2012.
In review. Submitted April 2012.
Many developers of audio signal processing strategies rely on objective measures of quality for initial evaluations of algorithms. As such, objective measures should be robust, and they should be able to predict quality accurately regardless of the dataset or testing conditions. Kates and Arehart have developed the Hearing Aid Speech Quality Index (HASQI) to predict the effects of noise, nonlinear distortion, and linear filtering on speech quality for both normal-hearing and hearing-impaired listeners, and they report very high performance with their training and testing datasets [Kates, J. and Arehart, K., Audio Eng. Soc., 58(5), 363-381 (2010)]. In order to investigate the generalizability of HASQI, we test its ability to predict normal-hearing listeners’ subjective quality ratings of a dataset on which it was not trained. This dataset is designed specifically to contain a wide range of distortions introduced by real-world noises which have been processed by some of the most common noise suppression algorithms in hearing aids. We show that HASQI achieves prediction performance comparable to PESQ, the standard for objective measures of quality, as well as some of the other measures in the literature. Furthermore, we identify areas of weakness and show that training can improve quantitative prediction.
A.S. Charles, P. Garrigues, and C.J. Rozell.
A common network architecture can efficiently implement diverse
sparsity-based inference problems.
2012.
In review. Submitted May 2012.
Recent signal processing developments have greatly expanded the range of sparsity based optimization functions for a variety of statistical models. Initial work in finding neurally plausible systems resulted in the locally competitive algorithms for solving basic sparsity inducing optimizations. Incorporating these new model types, however, would greatly increase the relevance of neural systems for solving regularized least-squares problems. In this work we show analytically that a wide range of sparse approximation problems can be solved in the LCA architecture, including approximate lp norms, modified l1 norms, re-weighted l1 and l2, the block l1 norm and classic Tikhonov regularization.
J.Y. Park, H.L. Yap, C.J. Rozell, and M.B. Wakin.
Concentration of measure for block diagonal matrices with
applications to compressive signal processing.
IEEE Transactions on Signal Processing, 59:5859-5875, December
2011.
[ .pdf ]
Theoretical analysis of randomized, compressive operators often depends on a concentration of measure inequality for the operator in question. Typically, such inequalities quantify the likelihood that a random matrix will preserve the norm of a signal after multiplication. Concentration of measure results are well-established for unstructured compressive matrices, populated with independent and identically distributed (i.i.d.) random entries. Many real-world acquisition systems, however, are subject to architectural constraints that make such matrices impractical. In this paper we derive concentration of measure bounds for two types of block diagonal compressive matrices, one in which the blocks along the main diagonal are random and independent, and one in which the blocks are random but equal. For both types of matrices, we show that the likelihood of norm preservation depends on certain properties of the signal being measured, but that for the best case signals, both types of block diagonal matrices can offer concentration performance on par with their unstructured, i.i.d. counterparts. We support our theoretical results with illustrative simulations as well as analytical and empirical investigations of several signal classes that are highly amenable to measurement using block diagonal matrices. We also discuss applications of these results in ensuring stable embeddings for various signal families and in establishing performance guarantees for
solving various signal processing tasks (such as detection and classification) directly in the compressed domain.
H.L. Yap and C.J. Rozell.
Stable Takens' embeddings for linear dynamical systems.
IEEE Transactions on Signal Processing, 59(10):4781-4794,
October 2011.
[ .pdf ]
Takens' Embedding Theorem remarkably established that concatenating M previous outputs of a dynamical system into a vector (called a delay coordinate map) can be a one-to-one mapping of a low-dimensional attractor from the system state-space.
However, Takens' theorem is fragile because even small imperfections can induce arbitrarily large errors in the attractor representation. We extend Takens' result to establish explicit, non-asymptotic sufficient conditions for a delay coordinate map to form a stable embedding in the restricted case of linear dynamical systems and observation functions.
Our work is inspired by the field of Compressive Sensing (CS), where results guarantee that low-dimensional signal families can be robustly reconstructed if they are stably embedded by a measurement operator.
However, in contrast to typical CS results, i) our sufficient conditions are independent of the size of the ambient state space (N), and ii) some system and measurement pairs have fundamental limits on the conditioning of the embedding (i.e., how close it is to an isometry), meaning that further measurements beyond some point add no further significant value.
We use several simple simulations to explore the conditions of the main results, including the tightness of the bounds and the convergence speed of the stable embedding.
A.S. Charles, B.A. Olshausen, and C.J. Rozell.
Learning sparse codes for hyperspectral imagery.
IEEE Journal of Selected Topics in Signal Processing,
5(5):963-978, September 2011.
[ .pdf ]
The spectral features in hyperspectral imagery (HSI) contain significant structure that, if properly characterized could enable more efficient data acquisition and improved data analysis. Because most pixels contain reflectances of just a few materials, we propose that a sparse coding model is well-matched to HSI data. Sparsity models consider each pixel as a combination of just a few elements from a larger dictionary, and this approach has proven effective in a wide range of applications. Furthermore, previous work has shown that optimal sparse coding dictionaries can be learned from a dataset with no other a priori information (in contrast to many HSI “endmember” discovery algorithms that assume the presence of pure spectra or side information).
We modified an existing unsupervised learning approach and applied it to HSI data (with significant ground truth labeling) to learn an optimal sparse coding dictionary. Using this learned dictionary, we demonstrate three main findings: i) the sparse coding model learns spectral signatures of materials in the scene and locally approximates nonlinear manifolds for individual materials, ii) this learned dictionary can be used to infer HSI-resolution data with very high accuracy from simulated imagery collected at multispectral-level resolution, and iii) this learned dictionary improves the performance of a supervised classification algorithm, both in terms of the classifier complexity and generalization from very small training sets.
C.J. Rozell, D.H Johnson, R.G. Baraniuk, and B.A. Olshausen.
Sparse coding via thresholding and local competition in neural
circuits.
Neural Computation, 20(10):2526-2563, October 2008.
[ .pdf ]
While evidence indicates that neural systems may be
employing sparse approximations to represent sensed
stimuli, the mechanisms underlying this ability are
not understood. We describe a local ly competitive
algorithm (LCA) that solves a collection of sparse
coding principles minimizing a weighted combination
of mean-squared error (MSE) and a coefficient cost
function. LCAs are designed to be implemented in a
dynamical system composed of many neuron-like
elements operating in parallel. These algorithms use
thresholding functions to induce local (usually
one-way) inhibitory competitions between nodes to
produce sparse representations. LCAs produce
coefficients with sparsity levels comparable to the
most popular centralized sparse coding algorithms
while being readily suited for neural
implementation. Addi- tionally, LCA coefficients for
video sequences demonstrate inertial properties that
are both qualitatively and quantitatively more
regular (i.e., smoother and more predictable) than
the coefficients produced by greedy algorithms.
We show that an Au nanoshell with a pH sensitive
molecular adsorbate functions as a standalone, all-optical nanoscale
pH meter that monitors its local environment through the
pH-dependent surface enhanced Raman scattering (SERS) spectra of the
adsorbate molecules. Moreover, we also show how the performance of
such a functional nanodevice can be quantitatively assessed. The
complex spectral output is reduced to a simple device characteristic
by application of a locally linear manifold approximation
algorithm. The average accuracy of the nano-“meter” was found to
be ± 0.10 pH units across its operating range.
C.J. Rozell and D.H. Johnson.
Analyzing the robustness of redundant population codes in sensory and
feature extraction systems.
Neurocomputing, 69(10-12):1215-1218, June 2006.
[ .pdf ]
Sensory systems often use groups of redundant neurons to represent
stimulus information both during transduction and population coding of
features. This redundancy makes the system more robust to corruption
in the representation. We approximate neural coding as a projection of
the stimulus onto a set of vectors, with the result encoded by spike
trains. We use the formalism of frame theory to quantify the inherent
noise reduction properties of such population codes. Additionally,
computing features from the stimulus signal can also be thought of as
projecting the coefficients of a sensory representation onto another
set of vectors specific to the feature of interest. The conditions
under which a combination of different features form a complete
representation for the stimulus signal can be found through a recent
extension to frame theory called “frames of subspaces.” We extend
the frame of subspaces theory to quantify the noise reduction
properties of a collection of redundant feature spaces.
C.J. Rozell and D.H. Johnson.
Examining methods for estimating mutual information in spiking neural
systems.
Neurocomputing, 65-66C:429-434, June 2005.
[ .pdf ]
Mutual information enjoys wide use in the computational neuroscience community for
analyzing spiking neural systems. Its direct calculation is difficult
because estimating the joint stimulus-response distribution requires a
prohibitive amount of data. Consequently, several techniques have
appeared for bounding mutual information that rely on less data. We
examine two upper bound techniques and find that they are either
unreliable or introduce strong assumptions about the neural code. We also
examine two lower bounds, showing that they can be very loose and
possibly bear little relation to the mutual information's actual value.
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Measuring information transfer in crayfish sustaining fiber spike
generators.
Biological Cybernetics, 90(2):89-97, February 2004.
[ .pdf ]
We present a method based
on information-theoretic distances for measuring the information
transfer efficiency of voltage to impulse encoders. In response to light
pulses, we simultaneously recorded the EPSP and spiking output of
crayfish sustaining fibers. To measure the distance between analog
EPSP responses, we developed a membrane noise model that accurately
captures stimulus-induced nonstationarities. By comparing the EPSP
and spike responses, we found encoding efficiencies on the order
of 10-4, with interesting dynamics occurring during initial
transients. A simple analog to point-process converter predicted the small
information transfer efficiencies and dynamic properties we measured.
Copyright held by Springer-Verlag. The original
publication is available at springerlink.com. DOI:10.1007/s00422-003-0458-y
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Information processing during transient responses in the crayfish
visual system.
Neurocomputing, 52-54:53-58, June 2003.
[ .pdf ]
We analyzed sustaining fiber
responses in the crayfish visual system to light pulses using information
processing techniques. The light pulse stimuli elicited a transient and
a steady-state component in the EPSP input and in the firing rate
of the spike train output. The overall information transfer of the
system was very low (10-4), with a sharp increase during the
transient portion of the response followed by a steady decrease. The
information transfer dynamics are consistent with a simple spike
generator model that depends explicitly on stimulus changes. The
present analysis also corroborates the observed light reflex behavior.
D.H. Johnson, I.N. Goodman, and C.J. Rozell.
Information theory and systems neuroscience.
In S. Grün and S. Rotter, editors, Analysis of parallel
spike trains. Springer-Verlag, 2010.
H.L. Yap, A. Charles, and C.J.
Rozell.
The restricted isometry property for echo state networks with
applications to sequence memory capacity.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
D. Sale, C.J. Rozell, J.K. Romberg, and A.D. Lanterman.
Compressive LADAR in realistic environments.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
S. Shapero, C. Rozell, A. Balavoine, and P. Hasler.
A scalable implementation of sparse approximation on a Field
Programmable Analog Array.
In IEEE Biomedical Circuits and Systems Conference (BioCAS), La
Jolla, CA, November 2011.
A. Kressner, D. Anderson, and C. Rozell.
Robustness of the hearing aid speech quality index (HASQI).
In IEEE Workshop on Applications of Signal Processing to Audio
and Acoustics (WASPAA), New Paltz, NY, October 2011.
[ .pdf ]
Objective measures of speech quality have been the subject of significant prior work, particularly in the areas of speech codecs and communication channels for normal-hearing listeners. One of the primary concerns of researchers in this area is how these metrics generalize to datasets or listener studies which are “unknown” to the measures. Another growing concern is how these metrics perform for the hearing-impaired community. Researchers working with the this community need to be able to predict how hearing-impaired listeners will perceive the quality of speech, as well as how they will perceive the quality of speech processed specifically by hear- ing aids. A relatively recent metric, the Hearing Aid Speech Quality Index (HASQI), is a model-based objective measure of quality developed in the context of hearing aids for normal-hearing and hearing-impaired listeners (Kates & Arehart, Journal of the Audio Engineering Society, 2010). As such, HASQI makes substantial progress on some of the generalization issues. However, HASQI has not been tested thus far on any datasets other than the one on which it was trained. The objective of this study is to demonstrate the robustness of HASQI in predicting subjective quality. We use an “unknown” dataset of noisy speech processed by noise suppression algorithms, along with a corresponding set of subjective quality scores from normal-hearing listeners, to demonstrate HASQI’s prediction performance. Furthermore, we compare HASQI’s performance with that of several other objective measures in order to provide a point of reference.
M.S. Asif, A. Charles, J. Romberg, and C. Rozell.
Estimation and dynamic updating of time-varying signals with sparse
variations.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Prague, Czech Republic, May 2011.
[ .pdf ]
This paper presents an algorithm for an l1-regularized Kalman filter. Given observations of a discrete-time linear dynamical system with sparse errors in the state evolution, we estimate the state sequence by solving an optimization algorithm that balances fidelity to the measurements (measured by the standard l2 norm) against the sparsity of the innovations (measured using the l1 norm). We also derive an efficient algorithm for updating the estimate as the system evolves. This dynamic updating algorithm uses a homotopy scheme that tracks the solution as new measurements are slowly worked into the system and old measurements are slowly removed. The effective cost of adding new measurements is a number of low-rank updates to the solution of a linear system of equations that is roughly proportional to the joint sparsity of all the innovations in the time interval of interest.
H.L. Yap, M.B. Wakin, and C.J. Rozell.
Stable manifold embeddings with operators satisfying the restricted
isometry property.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
Signals of interests can often be thought to come from a low dimensional signal model.
The exploitation of this fact has led to many recent interesting advances in signal processing, one notable example being in the field of compressive sensing (CS).
The literature on CS has established that many matrices satisfy the Restricted Isometry Property (RIP), which guarantees a stable (i.e., distance-preserving) embedding of a sparse signal model from an undersampled linear measurement system.
In this work, we study the stable embedding of manifold signal models using matrices that satisfy the RIP.
We show that by paying reasonable additional factors in the number of measurements, all matrices that satisfy the RIP can also be used (in conjunction with a random sign sequence) to obtain a stable embedding of a manifold.
H.L. Yap, A. Eftekhari, M.B. Wakin, and C.J. Rozell.
The restricted isometry property for block diagonal matrices.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In compressive sensing (CS), the Restricted Isometry Property (RIP) is a powerful condition on measurement operators which ensures robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms.
Early papers in CS showed that Gaussian random matrices satisfy the RIP with high probability, but such matrices are usually undesirable in practical applications due to storage limitations, computational considerations, or the mismatch of such matrices with certain measurement architectures.
To alleviate some or all of these difficulties, recent research efforts have focused on structured random matrices.
In this paper, we study block diagonal measurement matrices where each block on the main diagonal is itself a Gaussian random matrix.
The main result of this paper shows that such matrices can indeed satisfy the RIP but that the requisite number of measurements depends on the coherence of the basis in which the signals are sparse.
In the best case-for signals that are sparse in the frequency domain-these matrices perform nearly as well as dense Gaussian random matrices despite having many fewer nonzero entries.
A. Charles, M.S. Asif, J. Romberg, and C. Rozell.
Sparsity penalties in dynamical system estimation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In this work we address the problem of state estimation in dynamical systems using recent developments in compressive sensing and sparse approximation. We formulate the traditional Kalman filter as a one-step update optimization procedure which leads us to a more unified framework, useful for incorporating sparsity constraints. We introduce three combinations of two sparsity conditions (sparsity in the state and sparsity in the innovations) and write recursive optimization programs to estimate the state for each model. This paper is meant as an overview of different methods for incorporating sparsity into the dynamic model, a presentation of algorithms that unify the support and coefficient estimation, and a demonstration that these suboptimal schemes can actually show some performance improvements (either in estimation error or convergence time) over standard optimal methods that use an impoverished model.
A. Balavoine, C.J. Rozell, and J.K. Romberg.
Global convergence of the Locally Competitive Algorithm.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
The Locally Competitive Algorithm (LCA) is a continuous-time dynamical system designed to solve the problem of sparse approximation. This class of approximation problems plays an important role in producing state-of-the-art results in many signal processing and inverse problems, and implementing the LCA in analog VLSI may significantly improve the time and power necessary to solve these optimization programs. The goal of this paper is to analyze the dynamical behavior of the LCA system and guarantee its convergence and stability. We show that the LCA converges from any initial point. We also show that fixed points of the system are extrema of the sparse approximation objective function when designed for a certain class of sparsity-inducing cost penalty. In addition, we prove that under certain conditions on the solution, the LCA converges in a finite number of switches (i.e., node threshold crossings).
A. Charles, A.A. Kressner, and C.J.
Rozell.
Causal sparse decomposition of audio signals.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
Recent results have shown the utility of sparse coding for audio signals [citation]. While current inference methods can decompose audio signals, they require whole signals and are therefor ill suited for realtime applications that require causal processing. We propose a neurally inspired, causal, sparse inference scheme based on the Locally Competitive Algorithm (LCA) (Rozell et al. 2008) over a temporal-spectral neighborhood. We demonstrate that this causal inference scheme can achieve lower sparsity levels and better signal fidelity than current filter and threshold approaches. Additionally, for some regimes, the sparsity level approaches those of Matching Pursuit while still maintaining signal integrity.
H.L. Yap and C.J. Rozell.
Stable Takens' embedding for linear dynamical systems.
In Proceedings of the IEEE Conference on Decision and
Control, Atlanta, GA, December 2010.
Invited paper for session on Exploiting Sparsity and
Compressive Sensing in System Identification.
Takens' Embedding Theorem gives theoretical justification for the use of delay coordinate maps in characterizing and predicting nonlinear dynamical systems. However, in practice imperfections such as system and measurement noise may render these results unusable. In this paper, we consider conditions allowing for a stable version of Takens' Embedding Theorem in the restricted case of linear dynamical systems. Our work is inspired from results from the field of Compressive Sensing, where signals from a low-dimensional signal family residing in a high-dimensional space can be robustly recovered from compressive measurements only if the measurement form a stable embedding of the signal family. In particular, we show that a stable embedding of the attractor of the dynamical system is indeed possible
and give sufficient conditions on the number of delays and the observation function for the delay coordinate maps to be stabilized. In addition, we also show that when the attractor is an ellipse, the conditioning of the embedding is lower bounded by a positive constant dependent only on the dynamical system and not within control of the experimentalist. We illustrate our results with an example linear dynamical system converging to an elliptical attractor. Our analysis in this paper will give insights into stable Takens' Embedding of general dynamical systems.
C.J. Rozell and P. Garrigues.
Analog sparse approximation for compressed sensing recovery.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
Non-smooth convex optimization programs such as L1 minimization produce state-of-the-art results in many signal and image processing applications. Despite the progress in algorithms to solve these programs, they are still too computationally expensive for many real-time applications. Using recent results describing dynamical systems that efficiently solve these types of programs, we demonstrate through simulation that custom analog ICs implementations of this system could potentially perform compressed sensing recovery for real time applications approaching 500 KHz. Furthermore, we show that this architecture can implement several other optimization programs of recent interest, including reweighted L1 and group L1 minimization.
A. Charles and C.J. Rozell.
Sparse coding for spectral signatures in hyperspectral images.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
The growing use of hyperspectral imagery lead us to seek automated algorithms for extracting useful information about the scene. Recent work in sparse approximation has shown that unsupervised learning techniques can use example data to determine an efficient dictionary with few a priori assumptions. We apply this model to sample hyperspectral data and show that these techniques learn a dictionary that: 1) contains a meaningful spectral decomposition for hyperspectral imagery, 2) admit representations that are useful in determining properties and classifying materials in the scene, and 3) forms local approximations to the nonlinear manifold structure present in the actual data.
M.B. Wakin, J.Y. Park, H.L. Yap, and C.J. Rozell.
Concentration of measure for block diagonal measurement matrices.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Dallas, TX, March 2010.
[ .pdf ]
Concentration of measure inequalities are at the heart of much
theoretical analysis of randomized compressive operators. Though
commonly studied for dense matrices, in this paper we derive a
concentration of measure bound for block diagonal matrices where the
nonzero entries along the main diagonal blocks are i.i.d.
subgaussian random variables. Our main result states that the
concentration exponent, in the best case, scales as that for a fully
dense matrix. We also identify the role that the energy distribution
of the signal plays in distinguishing the best case from the worst.
We illustrate these phenomena with a series of experiments.
C.J. Rozell, H.L. Yap, J.Y. Park, and M.B. Wakin.
Concentration of measure for block diagonal matrices with repeated
blocks.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2010.
Invited paper.
[ .pdf ]
The theoretical analysis of randomized compressive operators often relies on the existence of a concentration of measure inequality for the operator of interest. Though commonly studied for unstructured, dense matrices, matrices with more structure are often of interest because they model constraints on the sensing system or allow more efficient system implementations. In this paper we derive a concentration of measure bound for block diagonal matrices where the nonzero entries along the main diagonal are a single repeated block of i.i.d. Gaussian random variables. Our main result states that the concentration exponent, in the best case, scales as that for a fully dense matrix. We also identify the role that the signal diversity plays in distinguishing the best and worst cases. Finally, we illustrate these phenomena with a series of experiments.
R.L. Ortman, C.J. Rozell, and D.H. Johnson.
Reconstruction of compressively sensed images via neurally plausible
local competitive algorithms.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), pages 476-479, Princeton, NJ, March 2008.
C.J. Rozell.
Distributed processing in frames for sparse approximation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2008.
Invited paper.
[ .pdf ]
Beyond signal processing applications, frames are also powerful tools
for modeling the sensing and information processing of many biological
and man-made systems that exhibit inherent redundancy. In many cases,
these systems are required to use distributed computational strategies
to analyze and process the sensory information. In this talk, I will
review the use of frames to model distributed sensing systems with a
particular focus on sensory neural systems. In light of the evidence
that many of these systems employ sparse codes, I will describe our
Locally Competitive Algorithms (LCAs) that use a dynamical system to
solve many sparse approximation problems. These LCAs employ a
parallel computational architecture with simple analog components. I
will show numerical simulation results for these systems and describe
their relationship to the many recently-proposed iterative
thresholding algorithms. Our LCA approach also demonstrates potential
advantages in coding time-varying signals (e.g., video) by reflecting
the smooth signal changes in smooth coefficient variations. Finally, I
will highlight some future directions where we hope to impact areas
such as efficient analog signal processing devices, fast discrete
approximation algorithms, and video processing and computer vision in
complex temporal environments.
C.J. Rozell, D.H. Johnson, R.G. Baraniuk, and B.A. Olshausen.
Locally competitive algorithms for sparse approximation.
In Proceedings of the International Conference on Image
Processing (ICIP), pages 169-172, San Antonio, TX, September 2007.
P. Casazza, G. Kutyniok, S. Li, and C.J. Rozell.
Modeling sensor networks with fusion frames.
In Proceedings of SPIE, Wavelets XII at SPIE Optics and
Photonics, volume 6701, pages 67011M-1 - 67011M-11, San Diego, CA, August
2007.
[ .pdf ]
The new notion of fusion frames will be presented in
this article. Fusion frames provide an extensive
framework not only to model sensor networks, but
also to serve as a means to improve robustness or
develop efficient and feasible reconstruction
algorithms. Fusion frames can be regarded as sets of
redundant subspaces each of which contains a
spanning set of local frame vectors, where the
subspaces have to satisfy special overlapping
properties. Main aspects of the theory of fusion
frames will be presented with a particular focus on
the design of sensor networks. New results on the
construction of Parseval fusion frames will also be
discussed.
C.J. Rozell and D.H. Johnson.
Power scheduling for wireless sensor and actuator networks.
In Proceedings of the International Conference on Information
Processing in Sensor Networks (IPSN), pages 470-478, Cambridge, MA, April
2007.
(Acceptance rate of 38/170˜22%.).
[ .pdf ]
We previously presented a model for some wireless sensor and actuator
network (WSAN) applications based on the vector space tools of frame
theory. In this WSAN model there is a weight associated to each
sensor-actuator link denoting the importance of that communication
link to the actuation fidelity. These weights were shown to be useful
in pruning away communication links to reduce the number of active
channels. Inspired by recent work in power scheduling for
decentralized estimation, we investigate the optimal allocation of
system resources for achieving a desired actuation fidelity. In this
scheme, each sensor acquires a noisy observation and sends a message
to a subset of actuators using an MQAM transmission strategy. The
message sent on each sensor-actuator communication link is quantized
with a variable number of bits, with the number of bits optimized to
minimize the total network power consumption subject to a constraint
on the actuation distortion. We show analytically and verify through
simulation that performing this optimal power scheduling can yield
significant power savings over communication strategies that use a
fixed number of bits on each communication link.
S.W. Bishnoi, C.S. Levin, C.J. Rozell, B.R. Johnson, D.H. Johnson, and N.J
Halas.
All-optical nanoscale pH meter: a plasmonic nanodevice with
quantifiable output.
In Proceedings of the Annual Meeting of the IEEE Lasers and
Electro-Optics Society (IEEE LEOS), Montreal, Canada, October 2006.
Invited paper.
C.J. Rozell and D.H. Johnson.
Evaluating local contributions to global performance in wireless
sensor and actuator networks.
Lecture Notes in Computer Science, 4026:1-16, June 2006.
Proceedings of the International Conference on Distributed
Computing in Sensor Systems (DCOSS), San Francisco, CA, June 2006.
[ .pdf ]
Wireless sensor networks are often studied with the goal of removing
information from the network as efficiently as possible. However,
when the application also includes an actuator network, it is
advantageous to determine actions in-network. In such settings,
optimizing the sensor node behavior with respect to sensor information
fidelity does not necessarily translate into optimum behavior in terms
of action fidelity. Inspired by neural systems, we present a model of
a sensor and actuator network based on the vector space tools of
frame theory that applies to applications analogous to reflex
behaviors in biological systems. Our analysis yields bounds on both
absolute and average actuation error that point directly to strategies
for limiting sensor communication based not only on local measurements
but also on a measure of how important each sensor-actuator link is to
the fidelity of the total actuation output.
C.J. Rozell, I.N. Goodman, and D.H. Johnson.
Feature-based information processing with selective attention.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Toulouse, France, May 2006.
[ .pdf ]
We present a simple but general model for feature-based
information processing with selective attention. We model
feature extraction as projections onto frames of subspaces,
which accounts for redundancies in the representations of
individual features as well as between features. To manage
limited resources, we use feedback attentional signals to
dynamically allocate system resources according to the
observed events. In our model, attention maximizes the
average information retained about all events weighted by
their relative priorities. We illustrate the model with a
simple system under a total bit constraint and discuss how
the organization of the feature extraction affects the
optimal bit allocation.
C.J. Rozell and D.H. Johnson.
Analysis of noise reduction in redundant expansions under distributed
processing requirements.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 185-188, Philadelphia, PA,
March 2005.
[ .pdf ]
We considered signal reconstruction with
redundant expansions under distributed processing in noisy environments.
Redundant expansions have the ability to reduce noise corrupting the
coefficients, but distributed processing schemes will not be able
to take full advantage of the redundancy present. We apply frame
theory and a generalization called “frames of subspaces” to find
conditions when distributed reconstruction suffers no loss in noise
reduction ability, and we bound performance loss in more general cases.
M.A. Lexa, C.J. Rozell, S. Sinanović, and D.H. Johnson.
To cooperate or not to cooperate: Detection strategies in sensor
networks.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 841-844, Montreal, Canada,
May 2004.
[ .pdf ]
This paper is an initial investigation into the following
question: Can cooperation among sensors in a sensor network improve
detection performance in a simple hypothesis test? We analyze a simple
cooperative system using the Kullback-Leibler (KL) discrimination distance
and a quantity known as the information transfer ratio which is a
ratio of KL distances. We discover that, asymptotically, gain over a
non-cooperative system depends on the conditional KL distance. We conclude
with an illustrative example which demonstrates that cooperation not
only significantly improves performance but can also degrade it.
C.J. Rozell and D. Manolakis.
Matched filter performance for unequal target and background
covariance matrices.
In Proceedings of the SPIE Defense and Security Symposium:
Algorithms and Technologies for Multispectral, Hyperspectral, and
Ultraspectral Imagery X, pages 109-117, Orlando, FL, April 2004.
Detection of military and civilian targets from
airborne platforms using hyperspectral imaging (HSI) sensors is of
great interest. Relative to multispectral sensing, hyperspectral
sensing can increase the detectability of targets by exploiting finer
detail in spectral signatures. A multitude of adaptive detection
algorithms have appeared in the literature or have found their way into
software packages and end-user systems. The most widely known among them
is the linear matched filter. However, despite its popularity, the
fact that the matched filter is used under conditions that deviate
from the implicit optimality assumptions has not been investigated.
M. Simoni, B. Broening, C. Rozell, C. Meek, and G. Wakefield.
A theoretical framework for electro-acoustic music.
In International Computer Music Conference (ICMC), Beijing,
China, 1999.
[ .pdf ]
A. Balavoine, J. Romberg, and C.J. Rozell.
Convergence and rate analysis of neural networks for sparse
approximation.
2012.
In review. Submitted July 2011. Revision posted March 27, 2012.
[ .pdf ]
We present an analysis of the Locally Competitive Algorithm (LCA), a Hopfield-style neural network that efficiently solves sparse approximation problems (e.g., approximating a vector from a dictionary using just a few non-zero coefficients). This class of problems plays a significant role in both theories of neural coding and applications in signal processing. However, the LCA lacks analysis of its convergence properties and previous results on neural networks for nonsmooth optimization do not apply to the specifics of the LCA architecture. We show that the LCA has desirable convergence properties, such as stability and global convergence to the optimum of the objective function when it is unique. Under some mild conditions, the support of the solution is also proven to be reached in finite time. Furthermore, some restrictions on the problem specifics allow us to characterize the convergence rate of the system by showing that the LCA converges exponentially fast with an analytically bounded convergence rate. We support our analysis with several illustrative simulations.
S. Shapero, C. Rozell, and P. Hasler.
Low power sparse approximation on reconfigurable analog hardware.
2012.
In review. Submitted November 2011.
Compressed sensing is an important optimization problem in signal and image processing applications. A Hopfield-Network-like analog system is proposed as a solution, using the Locally Competitive Algorithm (LCA) to solve an overcomplete l1 sparse approximation problem. A scalable system architecture using sub-threshold currents is described, including vector matrix multipliers (VMMs) and a nonlinear thresholder. A 4x6 nonlinear system is implemented on the RASP 2.9v chip, a Field Programmable Analog Array with directly programmable floating gate elements, allowing highly accurate VMMs. The circuit successfully reproduced the outputs of a digital optimization program, converging to within 4.8% RMS, and an optimization cost only 1.3% higher on average. The active circuit consumed 29μA of current at 2.4V, and converges on solutions in 240μs. A smaller 2x3 system is also implemented. Extrapolating the scaling trends to N=1000 node system, the Analog LCA compares favorably with State-of-the-Art digital solutions.
M. Zhu and C.J. Rozell.
Visual nonclassical receptive field effects emerge from sparse coding
in a dynamical system.
2012.
In review. Submitted April 2012.
Extensive electrophysiology studies have shown that many V1 simple cells receive contextual influence, where stimuli not part of the classical receptive field (CRF) can modulate the cell’s response to CRF stimuli. Mechanistic models of these non-classical receptive field (nCRF) effects do not address the role of these effects in the optimal coding or processing of natural stimuli. We demonstrate that a wide variety of nCRF effects are emergent properties of a single sparse coding model implemented in a neurally plausible network structure (requiring no tuning of any parameters to produce the various effects). In particular, this model produces both individual cells showing canonical nCRF effects, as well as the population diversity reported in the physiology literature. These results show that the sparse coding hypothesis, when coupled with a biophysically plausible implementation, can provide apparently disparate response properties with a unified interpretation based on the efficient coding of natural scenes.
A.A. Kressner, D.V. Anderson, and C.J. Rozell.
Evaluating the generalization of the hearing aid speech quality index
(HASQI).
2012.
In review. Submitted April 2012.
Many developers of audio signal processing strategies rely on objective measures of quality for initial evaluations of algorithms. As such, objective measures should be robust, and they should be able to predict quality accurately regardless of the dataset or testing conditions. Kates and Arehart have developed the Hearing Aid Speech Quality Index (HASQI) to predict the effects of noise, nonlinear distortion, and linear filtering on speech quality for both normal-hearing and hearing-impaired listeners, and they report very high performance with their training and testing datasets [Kates, J. and Arehart, K., Audio Eng. Soc., 58(5), 363-381 (2010)]. In order to investigate the generalizability of HASQI, we test its ability to predict normal-hearing listeners’ subjective quality ratings of a dataset on which it was not trained. This dataset is designed specifically to contain a wide range of distortions introduced by real-world noises which have been processed by some of the most common noise suppression algorithms in hearing aids. We show that HASQI achieves prediction performance comparable to PESQ, the standard for objective measures of quality, as well as some of the other measures in the literature. Furthermore, we identify areas of weakness and show that training can improve quantitative prediction.
J.Y. Park, H.L. Yap, C.J. Rozell, and M.B. Wakin.
Concentration of measure for block diagonal matrices with
applications to compressive signal processing.
IEEE Transactions on Signal Processing, 59:5859-5875, December
2011.
[ .pdf ]
Theoretical analysis of randomized, compressive operators often depends on a concentration of measure inequality for the operator in question. Typically, such inequalities quantify the likelihood that a random matrix will preserve the norm of a signal after multiplication. Concentration of measure results are well-established for unstructured compressive matrices, populated with independent and identically distributed (i.i.d.) random entries. Many real-world acquisition systems, however, are subject to architectural constraints that make such matrices impractical. In this paper we derive concentration of measure bounds for two types of block diagonal compressive matrices, one in which the blocks along the main diagonal are random and independent, and one in which the blocks are random but equal. For both types of matrices, we show that the likelihood of norm preservation depends on certain properties of the signal being measured, but that for the best case signals, both types of block diagonal matrices can offer concentration performance on par with their unstructured, i.i.d. counterparts. We support our theoretical results with illustrative simulations as well as analytical and empirical investigations of several signal classes that are highly amenable to measurement using block diagonal matrices. We also discuss applications of these results in ensuring stable embeddings for various signal families and in establishing performance guarantees for
solving various signal processing tasks (such as detection and classification) directly in the compressed domain.
H.L. Yap and C.J. Rozell.
Stable Takens' embeddings for linear dynamical systems.
IEEE Transactions on Signal Processing, 59(10):4781-4794,
October 2011.
[ .pdf ]
Takens' Embedding Theorem remarkably established that concatenating M previous outputs of a dynamical system into a vector (called a delay coordinate map) can be a one-to-one mapping of a low-dimensional attractor from the system state-space.
However, Takens' theorem is fragile because even small imperfections can induce arbitrarily large errors in the attractor representation. We extend Takens' result to establish explicit, non-asymptotic sufficient conditions for a delay coordinate map to form a stable embedding in the restricted case of linear dynamical systems and observation functions.
Our work is inspired by the field of Compressive Sensing (CS), where results guarantee that low-dimensional signal families can be robustly reconstructed if they are stably embedded by a measurement operator.
However, in contrast to typical CS results, i) our sufficient conditions are independent of the size of the ambient state space (N), and ii) some system and measurement pairs have fundamental limits on the conditioning of the embedding (i.e., how close it is to an isometry), meaning that further measurements beyond some point add no further significant value.
We use several simple simulations to explore the conditions of the main results, including the tightness of the bounds and the convergence speed of the stable embedding.
A.S. Charles, B.A. Olshausen, and C.J. Rozell.
Learning sparse codes for hyperspectral imagery.
IEEE Journal of Selected Topics in Signal Processing,
5(5):963-978, September 2011.
[ .pdf ]
The spectral features in hyperspectral imagery (HSI) contain significant structure that, if properly characterized could enable more efficient data acquisition and improved data analysis. Because most pixels contain reflectances of just a few materials, we propose that a sparse coding model is well-matched to HSI data. Sparsity models consider each pixel as a combination of just a few elements from a larger dictionary, and this approach has proven effective in a wide range of applications. Furthermore, previous work has shown that optimal sparse coding dictionaries can be learned from a dataset with no other a priori information (in contrast to many HSI “endmember” discovery algorithms that assume the presence of pure spectra or side information).
We modified an existing unsupervised learning approach and applied it to HSI data (with significant ground truth labeling) to learn an optimal sparse coding dictionary. Using this learned dictionary, we demonstrate three main findings: i) the sparse coding model learns spectral signatures of materials in the scene and locally approximates nonlinear manifolds for individual materials, ii) this learned dictionary can be used to infer HSI-resolution data with very high accuracy from simulated imagery collected at multispectral-level resolution, and iii) this learned dictionary improves the performance of a supervised classification algorithm, both in terms of the classifier complexity and generalization from very small training sets.
C.J. Rozell, D.H Johnson, R.G. Baraniuk, and B.A. Olshausen.
Sparse coding via thresholding and local competition in neural
circuits.
Neural Computation, 20(10):2526-2563, October 2008.
[ .pdf ]
While evidence indicates that neural systems may be
employing sparse approximations to represent sensed
stimuli, the mechanisms underlying this ability are
not understood. We describe a local ly competitive
algorithm (LCA) that solves a collection of sparse
coding principles minimizing a weighted combination
of mean-squared error (MSE) and a coefficient cost
function. LCAs are designed to be implemented in a
dynamical system composed of many neuron-like
elements operating in parallel. These algorithms use
thresholding functions to induce local (usually
one-way) inhibitory competitions between nodes to
produce sparse representations. LCAs produce
coefficients with sparsity levels comparable to the
most popular centralized sparse coding algorithms
while being readily suited for neural
implementation. Addi- tionally, LCA coefficients for
video sequences demonstrate inertial properties that
are both qualitatively and quantitatively more
regular (i.e., smoother and more predictable) than
the coefficients produced by greedy algorithms.
We show that an Au nanoshell with a pH sensitive
molecular adsorbate functions as a standalone, all-optical nanoscale
pH meter that monitors its local environment through the
pH-dependent surface enhanced Raman scattering (SERS) spectra of the
adsorbate molecules. Moreover, we also show how the performance of
such a functional nanodevice can be quantitatively assessed. The
complex spectral output is reduced to a simple device characteristic
by application of a locally linear manifold approximation
algorithm. The average accuracy of the nano-“meter” was found to
be ± 0.10 pH units across its operating range.
C.J. Rozell and D.H. Johnson.
Analyzing the robustness of redundant population codes in sensory and
feature extraction systems.
Neurocomputing, 69(10-12):1215-1218, June 2006.
[ .pdf ]
Sensory systems often use groups of redundant neurons to represent
stimulus information both during transduction and population coding of
features. This redundancy makes the system more robust to corruption
in the representation. We approximate neural coding as a projection of
the stimulus onto a set of vectors, with the result encoded by spike
trains. We use the formalism of frame theory to quantify the inherent
noise reduction properties of such population codes. Additionally,
computing features from the stimulus signal can also be thought of as
projecting the coefficients of a sensory representation onto another
set of vectors specific to the feature of interest. The conditions
under which a combination of different features form a complete
representation for the stimulus signal can be found through a recent
extension to frame theory called “frames of subspaces.” We extend
the frame of subspaces theory to quantify the noise reduction
properties of a collection of redundant feature spaces.
C.J. Rozell and D.H. Johnson.
Examining methods for estimating mutual information in spiking neural
systems.
Neurocomputing, 65-66C:429-434, June 2005.
[ .pdf ]
Mutual information enjoys wide use in the computational neuroscience community for
analyzing spiking neural systems. Its direct calculation is difficult
because estimating the joint stimulus-response distribution requires a
prohibitive amount of data. Consequently, several techniques have
appeared for bounding mutual information that rely on less data. We
examine two upper bound techniques and find that they are either
unreliable or introduce strong assumptions about the neural code. We also
examine two lower bounds, showing that they can be very loose and
possibly bear little relation to the mutual information's actual value.
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Measuring information transfer in crayfish sustaining fiber spike
generators.
Biological Cybernetics, 90(2):89-97, February 2004.
[ .pdf ]
We present a method based
on information-theoretic distances for measuring the information
transfer efficiency of voltage to impulse encoders. In response to light
pulses, we simultaneously recorded the EPSP and spiking output of
crayfish sustaining fibers. To measure the distance between analog
EPSP responses, we developed a membrane noise model that accurately
captures stimulus-induced nonstationarities. By comparing the EPSP
and spike responses, we found encoding efficiencies on the order
of 10-4, with interesting dynamics occurring during initial
transients. A simple analog to point-process converter predicted the small
information transfer efficiencies and dynamic properties we measured.
Copyright held by Springer-Verlag. The original
publication is available at springerlink.com. DOI:10.1007/s00422-003-0458-y
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Information processing during transient responses in the crayfish
visual system.
Neurocomputing, 52-54:53-58, June 2003.
[ .pdf ]
We analyzed sustaining fiber
responses in the crayfish visual system to light pulses using information
processing techniques. The light pulse stimuli elicited a transient and
a steady-state component in the EPSP input and in the firing rate
of the spike train output. The overall information transfer of the
system was very low (10-4), with a sharp increase during the
transient portion of the response followed by a steady decrease. The
information transfer dynamics are consistent with a simple spike
generator model that depends explicitly on stimulus changes. The
present analysis also corroborates the observed light reflex behavior.
D.H. Johnson, I.N. Goodman, and C.J. Rozell.
Information theory and systems neuroscience.
In S. Grün and S. Rotter, editors, Analysis of parallel
spike trains. Springer-Verlag, 2010.
H.L. Yap, A. Charles, and C.J.
Rozell.
The restricted isometry property for echo state networks with
applications to sequence memory capacity.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
D. Sale, C.J. Rozell, J.K. Romberg, and A.D. Lanterman.
Compressive LADAR in realistic environments.
In IEEE Statistical Signal Processing Workshop, Ann Arbor, MI,
August 2012.
S. Shapero, C. Rozell, A. Balavoine, and P. Hasler.
A scalable implementation of sparse approximation on a Field
Programmable Analog Array.
In IEEE Biomedical Circuits and Systems Conference (BioCAS), La
Jolla, CA, November 2011.
A. Kressner, D. Anderson, and C. Rozell.
Robustness of the hearing aid speech quality index (HASQI).
In IEEE Workshop on Applications of Signal Processing to Audio
and Acoustics (WASPAA), New Paltz, NY, October 2011.
[ .pdf ]
Objective measures of speech quality have been the subject of significant prior work, particularly in the areas of speech codecs and communication channels for normal-hearing listeners. One of the primary concerns of researchers in this area is how these metrics generalize to datasets or listener studies which are “unknown” to the measures. Another growing concern is how these metrics perform for the hearing-impaired community. Researchers working with the this community need to be able to predict how hearing-impaired listeners will perceive the quality of speech, as well as how they will perceive the quality of speech processed specifically by hear- ing aids. A relatively recent metric, the Hearing Aid Speech Quality Index (HASQI), is a model-based objective measure of quality developed in the context of hearing aids for normal-hearing and hearing-impaired listeners (Kates & Arehart, Journal of the Audio Engineering Society, 2010). As such, HASQI makes substantial progress on some of the generalization issues. However, HASQI has not been tested thus far on any datasets other than the one on which it was trained. The objective of this study is to demonstrate the robustness of HASQI in predicting subjective quality. We use an “unknown” dataset of noisy speech processed by noise suppression algorithms, along with a corresponding set of subjective quality scores from normal-hearing listeners, to demonstrate HASQI’s prediction performance. Furthermore, we compare HASQI’s performance with that of several other objective measures in order to provide a point of reference.
M.S. Asif, A. Charles, J. Romberg, and C. Rozell.
Estimation and dynamic updating of time-varying signals with sparse
variations.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Prague, Czech Republic, May 2011.
[ .pdf ]
This paper presents an algorithm for an l1-regularized Kalman filter. Given observations of a discrete-time linear dynamical system with sparse errors in the state evolution, we estimate the state sequence by solving an optimization algorithm that balances fidelity to the measurements (measured by the standard l2 norm) against the sparsity of the innovations (measured using the l1 norm). We also derive an efficient algorithm for updating the estimate as the system evolves. This dynamic updating algorithm uses a homotopy scheme that tracks the solution as new measurements are slowly worked into the system and old measurements are slowly removed. The effective cost of adding new measurements is a number of low-rank updates to the solution of a linear system of equations that is roughly proportional to the joint sparsity of all the innovations in the time interval of interest.
H.L. Yap, M.B. Wakin, and C.J. Rozell.
Stable manifold embeddings with operators satisfying the restricted
isometry property.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
Signals of interests can often be thought to come from a low dimensional signal model.
The exploitation of this fact has led to many recent interesting advances in signal processing, one notable example being in the field of compressive sensing (CS).
The literature on CS has established that many matrices satisfy the Restricted Isometry Property (RIP), which guarantees a stable (i.e., distance-preserving) embedding of a sparse signal model from an undersampled linear measurement system.
In this work, we study the stable embedding of manifold signal models using matrices that satisfy the RIP.
We show that by paying reasonable additional factors in the number of measurements, all matrices that satisfy the RIP can also be used (in conjunction with a random sign sequence) to obtain a stable embedding of a manifold.
H.L. Yap, A. Eftekhari, M.B. Wakin, and C.J. Rozell.
The restricted isometry property for block diagonal matrices.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In compressive sensing (CS), the Restricted Isometry Property (RIP) is a powerful condition on measurement operators which ensures robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms.
Early papers in CS showed that Gaussian random matrices satisfy the RIP with high probability, but such matrices are usually undesirable in practical applications due to storage limitations, computational considerations, or the mismatch of such matrices with certain measurement architectures.
To alleviate some or all of these difficulties, recent research efforts have focused on structured random matrices.
In this paper, we study block diagonal measurement matrices where each block on the main diagonal is itself a Gaussian random matrix.
The main result of this paper shows that such matrices can indeed satisfy the RIP but that the requisite number of measurements depends on the coherence of the basis in which the signals are sparse.
In the best case-for signals that are sparse in the frequency domain-these matrices perform nearly as well as dense Gaussian random matrices despite having many fewer nonzero entries.
A. Charles, M.S. Asif, J. Romberg, and C. Rozell.
Sparsity penalties in dynamical system estimation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In this work we address the problem of state estimation in dynamical systems using recent developments in compressive sensing and sparse approximation. We formulate the traditional Kalman filter as a one-step update optimization procedure which leads us to a more unified framework, useful for incorporating sparsity constraints. We introduce three combinations of two sparsity conditions (sparsity in the state and sparsity in the innovations) and write recursive optimization programs to estimate the state for each model. This paper is meant as an overview of different methods for incorporating sparsity into the dynamic model, a presentation of algorithms that unify the support and coefficient estimation, and a demonstration that these suboptimal schemes can actually show some performance improvements (either in estimation error or convergence time) over standard optimal methods that use an impoverished model.
A. Balavoine, C.J. Rozell, and J.K. Romberg.
Global convergence of the Locally Competitive Algorithm.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
The Locally Competitive Algorithm (LCA) is a continuous-time dynamical system designed to solve the problem of sparse approximation. This class of approximation problems plays an important role in producing state-of-the-art results in many signal processing and inverse problems, and implementing the LCA in analog VLSI may significantly improve the time and power necessary to solve these optimization programs. The goal of this paper is to analyze the dynamical behavior of the LCA system and guarantee its convergence and stability. We show that the LCA converges from any initial point. We also show that fixed points of the system are extrema of the sparse approximation objective function when designed for a certain class of sparsity-inducing cost penalty. In addition, we prove that under certain conditions on the solution, the LCA converges in a finite number of switches (i.e., node threshold crossings).
A. Charles, A.A. Kressner, and C.J.
Rozell.
Causal sparse decomposition of audio signals.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
Recent results have shown the utility of sparse coding for audio signals [citation]. While current inference methods can decompose audio signals, they require whole signals and are therefor ill suited for realtime applications that require causal processing. We propose a neurally inspired, causal, sparse inference scheme based on the Locally Competitive Algorithm (LCA) (Rozell et al. 2008) over a temporal-spectral neighborhood. We demonstrate that this causal inference scheme can achieve lower sparsity levels and better signal fidelity than current filter and threshold approaches. Additionally, for some regimes, the sparsity level approaches those of Matching Pursuit while still maintaining signal integrity.
H.L. Yap and C.J. Rozell.
Stable Takens' embedding for linear dynamical systems.
In Proceedings of the IEEE Conference on Decision and
Control, Atlanta, GA, December 2010.
Invited paper for session on Exploiting Sparsity and
Compressive Sensing in System Identification.
Takens' Embedding Theorem gives theoretical justification for the use of delay coordinate maps in characterizing and predicting nonlinear dynamical systems. However, in practice imperfections such as system and measurement noise may render these results unusable. In this paper, we consider conditions allowing for a stable version of Takens' Embedding Theorem in the restricted case of linear dynamical systems. Our work is inspired from results from the field of Compressive Sensing, where signals from a low-dimensional signal family residing in a high-dimensional space can be robustly recovered from compressive measurements only if the measurement form a stable embedding of the signal family. In particular, we show that a stable embedding of the attractor of the dynamical system is indeed possible
and give sufficient conditions on the number of delays and the observation function for the delay coordinate maps to be stabilized. In addition, we also show that when the attractor is an ellipse, the conditioning of the embedding is lower bounded by a positive constant dependent only on the dynamical system and not within control of the experimentalist. We illustrate our results with an example linear dynamical system converging to an elliptical attractor. Our analysis in this paper will give insights into stable Takens' Embedding of general dynamical systems.
C.J. Rozell and P. Garrigues.
Analog sparse approximation for compressed sensing recovery.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
Non-smooth convex optimization programs such as L1 minimization produce state-of-the-art results in many signal and image processing applications. Despite the progress in algorithms to solve these programs, they are still too computationally expensive for many real-time applications. Using recent results describing dynamical systems that efficiently solve these types of programs, we demonstrate through simulation that custom analog ICs implementations of this system could potentially perform compressed sensing recovery for real time applications approaching 500 KHz. Furthermore, we show that this architecture can implement several other optimization programs of recent interest, including reweighted L1 and group L1 minimization.
A. Charles and C.J. Rozell.
Sparse coding for spectral signatures in hyperspectral images.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
The growing use of hyperspectral imagery lead us to seek automated algorithms for extracting useful information about the scene. Recent work in sparse approximation has shown that unsupervised learning techniques can use example data to determine an efficient dictionary with few a priori assumptions. We apply this model to sample hyperspectral data and show that these techniques learn a dictionary that: 1) contains a meaningful spectral decomposition for hyperspectral imagery, 2) admit representations that are useful in determining properties and classifying materials in the scene, and 3) forms local approximations to the nonlinear manifold structure present in the actual data.
M.B. Wakin, J.Y. Park, H.L. Yap, and C.J. Rozell.
Concentration of measure for block diagonal measurement matrices.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Dallas, TX, March 2010.
[ .pdf ]
Concentration of measure inequalities are at the heart of much
theoretical analysis of randomized compressive operators. Though
commonly studied for dense matrices, in this paper we derive a
concentration of measure bound for block diagonal matrices where the
nonzero entries along the main diagonal blocks are i.i.d.
subgaussian random variables. Our main result states that the
concentration exponent, in the best case, scales as that for a fully
dense matrix. We also identify the role that the energy distribution
of the signal plays in distinguishing the best case from the worst.
We illustrate these phenomena with a series of experiments.
C.J. Rozell, H.L. Yap, J.Y. Park, and M.B. Wakin.
Concentration of measure for block diagonal matrices with repeated
blocks.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2010.
Invited paper.
[ .pdf ]
The theoretical analysis of randomized compressive operators often relies on the existence of a concentration of measure inequality for the operator of interest. Though commonly studied for unstructured, dense matrices, matrices with more structure are often of interest because they model constraints on the sensing system or allow more efficient system implementations. In this paper we derive a concentration of measure bound for block diagonal matrices where the nonzero entries along the main diagonal are a single repeated block of i.i.d. Gaussian random variables. Our main result states that the concentration exponent, in the best case, scales as that for a fully dense matrix. We also identify the role that the signal diversity plays in distinguishing the best and worst cases. Finally, we illustrate these phenomena with a series of experiments.
R.L. Ortman, C.J. Rozell, and D.H. Johnson.
Reconstruction of compressively sensed images via neurally plausible
local competitive algorithms.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), pages 476-479, Princeton, NJ, March 2008.
C.J. Rozell.
Distributed processing in frames for sparse approximation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2008.
Invited paper.
[ .pdf ]
Beyond signal processing applications, frames are also powerful tools
for modeling the sensing and information processing of many biological
and man-made systems that exhibit inherent redundancy. In many cases,
these systems are required to use distributed computational strategies
to analyze and process the sensory information. In this talk, I will
review the use of frames to model distributed sensing systems with a
particular focus on sensory neural systems. In light of the evidence
that many of these systems employ sparse codes, I will describe our
Locally Competitive Algorithms (LCAs) that use a dynamical system to
solve many sparse approximation problems. These LCAs employ a
parallel computational architecture with simple analog components. I
will show numerical simulation results for these systems and describe
their relationship to the many recently-proposed iterative
thresholding algorithms. Our LCA approach also demonstrates potential
advantages in coding time-varying signals (e.g., video) by reflecting
the smooth signal changes in smooth coefficient variations. Finally, I
will highlight some future directions where we hope to impact areas
such as efficient analog signal processing devices, fast discrete
approximation algorithms, and video processing and computer vision in
complex temporal environments.
C.J. Rozell, D.H. Johnson, R.G. Baraniuk, and B.A. Olshausen.
Locally competitive algorithms for sparse approximation.
In Proceedings of the International Conference on Image
Processing (ICIP), pages 169-172, San Antonio, TX, September 2007.
P. Casazza, G. Kutyniok, S. Li, and C.J. Rozell.
Modeling sensor networks with fusion frames.
In Proceedings of SPIE, Wavelets XII at SPIE Optics and
Photonics, volume 6701, pages 67011M-1 - 67011M-11, San Diego, CA, August
2007.
[ .pdf ]
The new notion of fusion frames will be presented in
this article. Fusion frames provide an extensive
framework not only to model sensor networks, but
also to serve as a means to improve robustness or
develop efficient and feasible reconstruction
algorithms. Fusion frames can be regarded as sets of
redundant subspaces each of which contains a
spanning set of local frame vectors, where the
subspaces have to satisfy special overlapping
properties. Main aspects of the theory of fusion
frames will be presented with a particular focus on
the design of sensor networks. New results on the
construction of Parseval fusion frames will also be
discussed.
C.J. Rozell and D.H. Johnson.
Power scheduling for wireless sensor and actuator networks.
In Proceedings of the International Conference on Information
Processing in Sensor Networks (IPSN), pages 470-478, Cambridge, MA, April
2007.
(Acceptance rate of 38/170˜22%.).
[ .pdf ]
We previously presented a model for some wireless sensor and actuator
network (WSAN) applications based on the vector space tools of frame
theory. In this WSAN model there is a weight associated to each
sensor-actuator link denoting the importance of that communication
link to the actuation fidelity. These weights were shown to be useful
in pruning away communication links to reduce the number of active
channels. Inspired by recent work in power scheduling for
decentralized estimation, we investigate the optimal allocation of
system resources for achieving a desired actuation fidelity. In this
scheme, each sensor acquires a noisy observation and sends a message
to a subset of actuators using an MQAM transmission strategy. The
message sent on each sensor-actuator communication link is quantized
with a variable number of bits, with the number of bits optimized to
minimize the total network power consumption subject to a constraint
on the actuation distortion. We show analytically and verify through
simulation that performing this optimal power scheduling can yield
significant power savings over communication strategies that use a
fixed number of bits on each communication link.
S.W. Bishnoi, C.S. Levin, C.J. Rozell, B.R. Johnson, D.H. Johnson, and N.J
Halas.
All-optical nanoscale pH meter: a plasmonic nanodevice with
quantifiable output.
In Proceedings of the Annual Meeting of the IEEE Lasers and
Electro-Optics Society (IEEE LEOS), Montreal, Canada, October 2006.
Invited paper.
C.J. Rozell and D.H. Johnson.
Evaluating local contributions to global performance in wireless
sensor and actuator networks.
Lecture Notes in Computer Science, 4026:1-16, June 2006.
Proceedings of the International Conference on Distributed
Computing in Sensor Systems (DCOSS), San Francisco, CA, June 2006.
[ .pdf ]
Wireless sensor networks are often studied with the goal of removing
information from the network as efficiently as possible. However,
when the application also includes an actuator network, it is
advantageous to determine actions in-network. In such settings,
optimizing the sensor node behavior with respect to sensor information
fidelity does not necessarily translate into optimum behavior in terms
of action fidelity. Inspired by neural systems, we present a model of
a sensor and actuator network based on the vector space tools of
frame theory that applies to applications analogous to reflex
behaviors in biological systems. Our analysis yields bounds on both
absolute and average actuation error that point directly to strategies
for limiting sensor communication based not only on local measurements
but also on a measure of how important each sensor-actuator link is to
the fidelity of the total actuation output.
C.J. Rozell, I.N. Goodman, and D.H. Johnson.
Feature-based information processing with selective attention.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Toulouse, France, May 2006.
[ .pdf ]
We present a simple but general model for feature-based
information processing with selective attention. We model
feature extraction as projections onto frames of subspaces,
which accounts for redundancies in the representations of
individual features as well as between features. To manage
limited resources, we use feedback attentional signals to
dynamically allocate system resources according to the
observed events. In our model, attention maximizes the
average information retained about all events weighted by
their relative priorities. We illustrate the model with a
simple system under a total bit constraint and discuss how
the organization of the feature extraction affects the
optimal bit allocation.
C.J. Rozell and D.H. Johnson.
Analysis of noise reduction in redundant expansions under distributed
processing requirements.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 185-188, Philadelphia, PA,
March 2005.
[ .pdf ]
We considered signal reconstruction with
redundant expansions under distributed processing in noisy environments.
Redundant expansions have the ability to reduce noise corrupting the
coefficients, but distributed processing schemes will not be able
to take full advantage of the redundancy present. We apply frame
theory and a generalization called “frames of subspaces” to find
conditions when distributed reconstruction suffers no loss in noise
reduction ability, and we bound performance loss in more general cases.
M.A. Lexa, C.J. Rozell, S. Sinanović, and D.H. Johnson.
To cooperate or not to cooperate: Detection strategies in sensor
networks.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 841-844, Montreal, Canada,
May 2004.
[ .pdf ]
This paper is an initial investigation into the following
question: Can cooperation among sensors in a sensor network improve
detection performance in a simple hypothesis test? We analyze a simple
cooperative system using the Kullback-Leibler (KL) discrimination distance
and a quantity known as the information transfer ratio which is a
ratio of KL distances. We discover that, asymptotically, gain over a
non-cooperative system depends on the conditional KL distance. We conclude
with an illustrative example which demonstrates that cooperation not
only significantly improves performance but can also degrade it.
C.J. Rozell and D. Manolakis.
Matched filter performance for unequal target and background
covariance matrices.
In Proceedings of the SPIE Defense and Security Symposium:
Algorithms and Technologies for Multispectral, Hyperspectral, and
Ultraspectral Imagery X, pages 109-117, Orlando, FL, April 2004.
Detection of military and civilian targets from
airborne platforms using hyperspectral imaging (HSI) sensors is of
great interest. Relative to multispectral sensing, hyperspectral
sensing can increase the detectability of targets by exploiting finer
detail in spectral signatures. A multitude of adaptive detection
algorithms have appeared in the literature or have found their way into
software packages and end-user systems. The most widely known among them
is the linear matched filter. However, despite its popularity, the
fact that the matched filter is used under conditions that deviate
from the implicit optimality assumptions has not been investigated.
M. Simoni, B. Broening, C. Rozell, C. Meek, and G. Wakefield.
A theoretical framework for electro-acoustic music.
In International Computer Music Conference (ICMC), Beijing,
China, 1999.
[ .pdf ]
J.Y. Park, H.L. Yap, C.J. Rozell, and M.B. Wakin.
Concentration of measure for block diagonal matrices with
applications to compressive signal processing.
IEEE Transactions on Signal Processing, 59:5859-5875, December
2011.
[ .pdf ]
Theoretical analysis of randomized, compressive operators often depends on a concentration of measure inequality for the operator in question. Typically, such inequalities quantify the likelihood that a random matrix will preserve the norm of a signal after multiplication. Concentration of measure results are well-established for unstructured compressive matrices, populated with independent and identically distributed (i.i.d.) random entries. Many real-world acquisition systems, however, are subject to architectural constraints that make such matrices impractical. In this paper we derive concentration of measure bounds for two types of block diagonal compressive matrices, one in which the blocks along the main diagonal are random and independent, and one in which the blocks are random but equal. For both types of matrices, we show that the likelihood of norm preservation depends on certain properties of the signal being measured, but that for the best case signals, both types of block diagonal matrices can offer concentration performance on par with their unstructured, i.i.d. counterparts. We support our theoretical results with illustrative simulations as well as analytical and empirical investigations of several signal classes that are highly amenable to measurement using block diagonal matrices. We also discuss applications of these results in ensuring stable embeddings for various signal families and in establishing performance guarantees for
solving various signal processing tasks (such as detection and classification) directly in the compressed domain.
S. Shapero, C. Rozell, and P. Hasler.
Low power sparse approximation on reconfigurable analog hardware.
November 2011.
Submitted.
Compressed sensing is an important optimization problem in signal and image processing applications. A Hopfield-Network-like analog system is proposed as a solution, using the Locally Competitive Algorithm (LCA) to solve an overcomplete l1 sparse approximation problem. A scalable system architecture using sub-threshold currents is described, including vector matrix multipliers (VMMs) and a nonlinear thresholder. A 4x6 nonlinear system is implemented on the RASP 2.9v chip, a Field Programmable Analog Array with directly programmable floating gate elements, allowing highly accurate VMMs. The circuit successfully reproduced the outputs of a digital optimization program, converging to within 4.8% RMS, and an optimization cost only 1.3% higher on average. The active circuit consumed 29μA of current at 2.4V, and converges on solutions in 240μs. A smaller 2x3 system is also implemented. Extrapolating the scaling trends to N=1000 node system, the Analog LCA compares favorably with State-of-the-Art digital solutions
A. Charles, P. Garrigues, and C.J. Rozell.
Analog sparse approximation with applications to compressed sensing.
November 2011.
Submitted.
[ .pdf ]
Recent research has shown that performance in signal processing tasks can often be significantly improved by using signal models based on sparse representations, where a signal is approximated using a small number of elements from a fixed dictionary. Unfortunately, inference in this model involves solving non-smooth optimization problems that are computationally expensive. While significant efforts have focused on developing digital algorithms specifically for this problem, these algorithms are inappropriate for many applications because of the time and power requirements necessary to solve large optimization problems. Based on recent work in computational neuroscience, we explore the potential advantages of continuous time dynamical systems for solving sparse approximation problems if they were implemented in analog VLSI. Specifically, in the simulated task of recovering synthetic and MRI data acquired via compressive sensing techniques, we show that these systems can potentially perform recovery at time scales of 10-20μs, supporting datarates of 50-100 kHz (orders of magnitude faster that digital algorithms). Furthermore, we show analytically that a wide range of sparse approximation problems can be solved in the same basic architecture, including approximate p norms, modified 1 norms, re-weighted 1 and 2, the block 1 norm and classic Tikhonov regularization.
H.L. Yap and C.J. Rozell.
Stable Takens' embeddings for linear dynamical systems.
IEEE Transactions on Signal Processing, 59(10):4781-4794,
October 2011.
[ .pdf ]
Takens' Embedding Theorem remarkably established that concatenating M previous outputs of a dynamical system into a vector (called a delay coordinate map) can be a one-to-one mapping of a low-dimensional attractor from the system state-space.
However, Takens' theorem is fragile because even small imperfections can induce arbitrarily large errors in the attractor representation. We extend Takens' result to establish explicit, non-asymptotic sufficient conditions for a delay coordinate map to form a stable embedding in the restricted case of linear dynamical systems and observation functions.
Our work is inspired by the field of Compressive Sensing (CS), where results guarantee that low-dimensional signal families can be robustly reconstructed if they are stably embedded by a measurement operator.
However, in contrast to typical CS results, i) our sufficient conditions are independent of the size of the ambient state space (N), and ii) some system and measurement pairs have fundamental limits on the conditioning of the embedding (i.e., how close it is to an isometry), meaning that further measurements beyond some point add no further significant value.
We use several simple simulations to explore the conditions of the main results, including the tightness of the bounds and the convergence speed of the stable embedding.
A.S. Charles, B.A. Olshausen, and C.J. Rozell.
Learning sparse codes for hyperspectral imagery.
IEEE Journal of Selected Topics in Signal Processing,
5(5):963-978, September 2011.
[ .pdf ]
The spectral features in hyperspectral imagery (HSI) contain significant structure that, if properly characterized could enable more efficient data acquisition and improved data analysis. Because most pixels contain reflectances of just a few materials, we propose that a sparse coding model is well-matched to HSI data. Sparsity models consider each pixel as a combination of just a few elements from a larger dictionary, and this approach has proven effective in a wide range of applications. Furthermore, previous work has shown that optimal sparse coding dictionaries can be learned from a dataset with no other a priori information (in contrast to many HSI “endmember” discovery algorithms that assume the presence of pure spectra or side information).
We modified an existing unsupervised learning approach and applied it to HSI data (with significant ground truth labeling) to learn an optimal sparse coding dictionary. Using this learned dictionary, we demonstrate three main findings: i) the sparse coding model learns spectral signatures of materials in the scene and locally approximates nonlinear manifolds for individual materials, ii) this learned dictionary can be used to infer HSI-resolution data with very high accuracy from simulated imagery collected at multispectral-level resolution, and iii) this learned dictionary improves the performance of a supervised classification algorithm, both in terms of the classifier complexity and generalization from very small training sets.
A. Balavoine, J. Romberg, and C.J. Rozell.
Convergence and rate analysis of neural networks for sparse
approximation.
July 2011.
Submitted. Revision posted March 27, 2012.
[ .pdf ]
We present an analysis of the Locally Competitive Algorithm (LCA), a Hopfield-style neural network that efficiently solves sparse approximation problems (e.g., approximating a vector from a dictionary using just a few non-zero coefficients). This class of problems plays a significant role in both theories of neural coding and applications in signal processing. However, the LCA lacks analysis of its convergence properties and previous results on neural networks for nonsmooth optimization do not apply to the specifics of the LCA architecture. We show that the LCA has desirable convergence properties, such as stability and global convergence to the optimum of the objective function when it is unique. Under some mild conditions, the support of the solution is also proven to be reached in finite time. Furthermore, some restrictions on the problem specifics allow us to characterize the convergence rate of the system by showing that the LCA converges exponentially fast with an analytically bounded convergence rate. We support our analysis with several illustrative simulations.
C.J. Rozell, D.H Johnson, R.G. Baraniuk, and B.A. Olshausen.
Sparse coding via thresholding and local competition in neural
circuits.
Neural Computation, 20(10):2526-2563, October 2008.
[ .pdf ]
While evidence indicates that neural systems may be
employing sparse approximations to represent sensed
stimuli, the mechanisms underlying this ability are
not understood. We describe a local ly competitive
algorithm (LCA) that solves a collection of sparse
coding principles minimizing a weighted combination
of mean-squared error (MSE) and a coefficient cost
function. LCAs are designed to be implemented in a
dynamical system composed of many neuron-like
elements operating in parallel. These algorithms use
thresholding functions to induce local (usually
one-way) inhibitory competitions between nodes to
produce sparse representations. LCAs produce
coefficients with sparsity levels comparable to the
most popular centralized sparse coding algorithms
while being readily suited for neural
implementation. Addi- tionally, LCA coefficients for
video sequences demonstrate inertial properties that
are both qualitatively and quantitatively more
regular (i.e., smoother and more predictable) than
the coefficients produced by greedy algorithms.
We show that an Au nanoshell with a pH sensitive
molecular adsorbate functions as a standalone, all-optical nanoscale
pH meter that monitors its local environment through the
pH-dependent surface enhanced Raman scattering (SERS) spectra of the
adsorbate molecules. Moreover, we also show how the performance of
such a functional nanodevice can be quantitatively assessed. The
complex spectral output is reduced to a simple device characteristic
by application of a locally linear manifold approximation
algorithm. The average accuracy of the nano-“meter” was found to
be ± 0.10 pH units across its operating range.
C.J. Rozell and D.H. Johnson.
Analyzing the robustness of redundant population codes in sensory and
feature extraction systems.
Neurocomputing, 69(10-12):1215-1218, June 2006.
[ .pdf ]
Sensory systems often use groups of redundant neurons to represent
stimulus information both during transduction and population coding of
features. This redundancy makes the system more robust to corruption
in the representation. We approximate neural coding as a projection of
the stimulus onto a set of vectors, with the result encoded by spike
trains. We use the formalism of frame theory to quantify the inherent
noise reduction properties of such population codes. Additionally,
computing features from the stimulus signal can also be thought of as
projecting the coefficients of a sensory representation onto another
set of vectors specific to the feature of interest. The conditions
under which a combination of different features form a complete
representation for the stimulus signal can be found through a recent
extension to frame theory called “frames of subspaces.” We extend
the frame of subspaces theory to quantify the noise reduction
properties of a collection of redundant feature spaces.
C.J. Rozell and D.H. Johnson.
Examining methods for estimating mutual information in spiking neural
systems.
Neurocomputing, 65-66C:429-434, June 2005.
[ .pdf ]
Mutual information enjoys wide use in the computational neuroscience community for
analyzing spiking neural systems. Its direct calculation is difficult
because estimating the joint stimulus-response distribution requires a
prohibitive amount of data. Consequently, several techniques have
appeared for bounding mutual information that rely on less data. We
examine two upper bound techniques and find that they are either
unreliable or introduce strong assumptions about the neural code. We also
examine two lower bounds, showing that they can be very loose and
possibly bear little relation to the mutual information's actual value.
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Measuring information transfer in crayfish sustaining fiber spike
generators.
Biological Cybernetics, 90(2):89-97, February 2004.
[ .pdf ]
We present a method based
on information-theoretic distances for measuring the information
transfer efficiency of voltage to impulse encoders. In response to light
pulses, we simultaneously recorded the EPSP and spiking output of
crayfish sustaining fibers. To measure the distance between analog
EPSP responses, we developed a membrane noise model that accurately
captures stimulus-induced nonstationarities. By comparing the EPSP
and spike responses, we found encoding efficiencies on the order
of 10-4, with interesting dynamics occurring during initial
transients. A simple analog to point-process converter predicted the small
information transfer efficiencies and dynamic properties we measured.
Copyright held by Springer-Verlag. The original
publication is available at springerlink.com. DOI:10.1007/s00422-003-0458-y
C.J. Rozell, D.H. Johnson, and R.M. Glantz.
Information processing during transient responses in the crayfish
visual system.
Neurocomputing, 52-54:53-58, June 2003.
[ .pdf ]
We analyzed sustaining fiber
responses in the crayfish visual system to light pulses using information
processing techniques. The light pulse stimuli elicited a transient and
a steady-state component in the EPSP input and in the firing rate
of the spike train output. The overall information transfer of the
system was very low (10-4), with a sharp increase during the
transient portion of the response followed by a steady decrease. The
information transfer dynamics are consistent with a simple spike
generator model that depends explicitly on stimulus changes. The
present analysis also corroborates the observed light reflex behavior.
D.H. Johnson, I.N. Goodman, and C.J. Rozell.
Information theory and systems neuroscience.
In S. Grün and S. Rotter, editors, Analysis of parallel
spike trains. Springer-Verlag, 2010.
S. Shapero, C. Rozell, A. Balavoine, and P. Hasler.
A scalable implementation of sparse approximation on a Field
Programmable Analog Array.
In IEEE Biomedical Circuits and Systems Conference (BioCAS), La
Jolla, CA, November 2011.
A. Kressner, D. Anderson, and C. Rozell.
Robustness of the hearing aid speech quality index (HASQI).
In IEEE Workshop on Applications of Signal Processing to Audio
and Acoustics (WASPAA), New Paltz, NY, October 2011.
[ .pdf ]
Objective measures of speech quality have been the subject of significant prior work, particularly in the areas of speech codecs and communication channels for normal-hearing listeners. One of the primary concerns of researchers in this area is how these metrics generalize to datasets or listener studies which are “unknown” to the measures. Another growing concern is how these metrics perform for the hearing-impaired community. Researchers working with the this community need to be able to predict how hearing-impaired listeners will perceive the quality of speech, as well as how they will perceive the quality of speech processed specifically by hear- ing aids. A relatively recent metric, the Hearing Aid Speech Quality Index (HASQI), is a model-based objective measure of quality developed in the context of hearing aids for normal-hearing and hearing-impaired listeners (Kates & Arehart, Journal of the Audio Engineering Society, 2010). As such, HASQI makes substantial progress on some of the generalization issues. However, HASQI has not been tested thus far on any datasets other than the one on which it was trained. The objective of this study is to demonstrate the robustness of HASQI in predicting subjective quality. We use an “unknown” dataset of noisy speech processed by noise suppression algorithms, along with a corresponding set of subjective quality scores from normal-hearing listeners, to demonstrate HASQI’s prediction performance. Furthermore, we compare HASQI’s performance with that of several other objective measures in order to provide a point of reference.
M.S. Asif, A. Charles, J. Romberg, and C. Rozell.
Estimation and dynamic updating of time-varying signals with sparse
variations.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Prague, Czech Republic, May 2011.
[ .pdf ]
This paper presents an algorithm for an l1-regularized Kalman filter. Given observations of a discrete-time linear dynamical system with sparse errors in the state evolution, we estimate the state sequence by solving an optimization algorithm that balances fidelity to the measurements (measured by the standard l2 norm) against the sparsity of the innovations (measured using the l1 norm). We also derive an efficient algorithm for updating the estimate as the system evolves. This dynamic updating algorithm uses a homotopy scheme that tracks the solution as new measurements are slowly worked into the system and old measurements are slowly removed. The effective cost of adding new measurements is a number of low-rank updates to the solution of a linear system of equations that is roughly proportional to the joint sparsity of all the innovations in the time interval of interest.
H.L. Yap, M.B. Wakin, and C.J. Rozell.
Stable manifold embeddings with operators satisfying the restricted
isometry property.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
Signals of interests can often be thought to come from a low dimensional signal model.
The exploitation of this fact has led to many recent interesting advances in signal processing, one notable example being in the field of compressive sensing (CS).
The literature on CS has established that many matrices satisfy the Restricted Isometry Property (RIP), which guarantees a stable (i.e., distance-preserving) embedding of a sparse signal model from an undersampled linear measurement system.
In this work, we study the stable embedding of manifold signal models using matrices that satisfy the RIP.
We show that by paying reasonable additional factors in the number of measurements, all matrices that satisfy the RIP can also be used (in conjunction with a random sign sequence) to obtain a stable embedding of a manifold.
H.L. Yap, A. Eftekhari, M.B. Wakin, and C.J. Rozell.
The restricted isometry property for block diagonal matrices.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In compressive sensing (CS), the Restricted Isometry Property (RIP) is a powerful condition on measurement operators which ensures robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms.
Early papers in CS showed that Gaussian random matrices satisfy the RIP with high probability, but such matrices are usually undesirable in practical applications due to storage limitations, computational considerations, or the mismatch of such matrices with certain measurement architectures.
To alleviate some or all of these difficulties, recent research efforts have focused on structured random matrices.
In this paper, we study block diagonal measurement matrices where each block on the main diagonal is itself a Gaussian random matrix.
The main result of this paper shows that such matrices can indeed satisfy the RIP but that the requisite number of measurements depends on the coherence of the basis in which the signals are sparse.
In the best case-for signals that are sparse in the frequency domain-these matrices perform nearly as well as dense Gaussian random matrices despite having many fewer nonzero entries.
A. Charles, M.S. Asif, J. Romberg, and C. Rozell.
Sparsity penalties in dynamical system estimation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Baltimore, MD, March 2011.
[ .pdf ]
In this work we address the problem of state estimation in dynamical systems using recent developments in compressive sensing and sparse approximation. We formulate the traditional Kalman filter as a one-step update optimization procedure which leads us to a more unified framework, useful for incorporating sparsity constraints. We introduce three combinations of two sparsity conditions (sparsity in the state and sparsity in the innovations) and write recursive optimization programs to estimate the state for each model. This paper is meant as an overview of different methods for incorporating sparsity into the dynamic model, a presentation of algorithms that unify the support and coefficient estimation, and a demonstration that these suboptimal schemes can actually show some performance improvements (either in estimation error or convergence time) over standard optimal methods that use an impoverished model.
A. Balavoine, C.J. Rozell, and J.K. Romberg.
Global convergence of the Locally Competitive Algorithm.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
The Locally Competitive Algorithm (LCA) is a continuous-time dynamical system designed to solve the problem of sparse approximation. This class of approximation problems plays an important role in producing state-of-the-art results in many signal processing and inverse problems, and implementing the LCA in analog VLSI may significantly improve the time and power necessary to solve these optimization programs. The goal of this paper is to analyze the dynamical behavior of the LCA system and guarantee its convergence and stability. We show that the LCA converges from any initial point. We also show that fixed points of the system are extrema of the sparse approximation objective function when designed for a certain class of sparsity-inducing cost penalty. In addition, we prove that under certain conditions on the solution, the LCA converges in a finite number of switches (i.e., node threshold crossings).
A. Charles, A.A. Kressner, and C.J.
Rozell.
Causal sparse decomposition of audio signals.
In Proceedings of the IEEE Digital Signal Processing (DSP)
Workshop, Sedona, AZ, January 2011.
Recent results have shown the utility of sparse coding for audio signals [citation]. While current inference methods can decompose audio signals, they require whole signals and are therefor ill suited for realtime applications that require causal processing. We propose a neurally inspired, causal, sparse inference scheme based on the Locally Competitive Algorithm (LCA) (Rozell et al. 2008) over a temporal-spectral neighborhood. We demonstrate that this causal inference scheme can achieve lower sparsity levels and better signal fidelity than current filter and threshold approaches. Additionally, for some regimes, the sparsity level approaches those of Matching Pursuit while still maintaining signal integrity.
H.L. Yap and C.J. Rozell.
Stable Takens' embedding for linear dynamical systems.
In Proceedings of the IEEE Conference on Decision and
Control, Atlanta, GA, December 2010.
Invited paper for session on Exploiting Sparsity and
Compressive Sensing in System Identification.
Takens' Embedding Theorem gives theoretical justification for the use of delay coordinate maps in characterizing and predicting nonlinear dynamical systems. However, in practice imperfections such as system and measurement noise may render these results unusable. In this paper, we consider conditions allowing for a stable version of Takens' Embedding Theorem in the restricted case of linear dynamical systems. Our work is inspired from results from the field of Compressive Sensing, where signals from a low-dimensional signal family residing in a high-dimensional space can be robustly recovered from compressive measurements only if the measurement form a stable embedding of the signal family. In particular, we show that a stable embedding of the attractor of the dynamical system is indeed possible
and give sufficient conditions on the number of delays and the observation function for the delay coordinate maps to be stabilized. In addition, we also show that when the attractor is an ellipse, the conditioning of the embedding is lower bounded by a positive constant dependent only on the dynamical system and not within control of the experimentalist. We illustrate our results with an example linear dynamical system converging to an elliptical attractor. Our analysis in this paper will give insights into stable Takens' Embedding of general dynamical systems.
C.J. Rozell and P. Garrigues.
Analog sparse approximation for compressed sensing recovery.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
Non-smooth convex optimization programs such as L1 minimization produce state-of-the-art results in many signal and image processing applications. Despite the progress in algorithms to solve these programs, they are still too computationally expensive for many real-time applications. Using recent results describing dynamical systems that efficiently solve these types of programs, we demonstrate through simulation that custom analog ICs implementations of this system could potentially perform compressed sensing recovery for real time applications approaching 500 KHz. Furthermore, we show that this architecture can implement several other optimization programs of recent interest, including reweighted L1 and group L1 minimization.
A. Charles and C.J. Rozell.
Sparse coding for spectral signatures in hyperspectral images.
In Proceedings of the Asilomar Conference on Signals, Systems,
and Computers, Pacific Grove, CA, November 2010.
The growing use of hyperspectral imagery lead us to seek automated algorithms for extracting useful information about the scene. Recent work in sparse approximation has shown that unsupervised learning techniques can use example data to determine an efficient dictionary with few a priori assumptions. We apply this model to sample hyperspectral data and show that these techniques learn a dictionary that: 1) contains a meaningful spectral decomposition for hyperspectral imagery, 2) admit representations that are useful in determining properties and classifying materials in the scene, and 3) forms local approximations to the nonlinear manifold structure present in the actual data.
M.B. Wakin, J.Y. Park, H.L. Yap, and C.J. Rozell.
Concentration of measure for block diagonal measurement matrices.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Dallas, TX, March 2010.
[ .pdf ]
Concentration of measure inequalities are at the heart of much
theoretical analysis of randomized compressive operators. Though
commonly studied for dense matrices, in this paper we derive a
concentration of measure bound for block diagonal matrices where the
nonzero entries along the main diagonal blocks are i.i.d.
subgaussian random variables. Our main result states that the
concentration exponent, in the best case, scales as that for a fully
dense matrix. We also identify the role that the energy distribution
of the signal plays in distinguishing the best case from the worst.
We illustrate these phenomena with a series of experiments.
C.J. Rozell, H.L. Yap, J.Y. Park, and M.B. Wakin.
Concentration of measure for block diagonal matrices with repeated
blocks.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2010.
Invited paper.
[ .pdf ]
The theoretical analysis of randomized compressive operators often relies on the existence of a concentration of measure inequality for the operator of interest. Though commonly studied for unstructured, dense matrices, matrices with more structure are often of interest because they model constraints on the sensing system or allow more efficient system implementations. In this paper we derive a concentration of measure bound for block diagonal matrices where the nonzero entries along the main diagonal are a single repeated block of i.i.d. Gaussian random variables. Our main result states that the concentration exponent, in the best case, scales as that for a fully dense matrix. We also identify the role that the signal diversity plays in distinguishing the best and worst cases. Finally, we illustrate these phenomena with a series of experiments.
R.L. Ortman, C.J. Rozell, and D.H. Johnson.
Reconstruction of compressively sensed images via neurally plausible
local competitive algorithms.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), pages 476-479, Princeton, NJ, March 2008.
C.J. Rozell.
Distributed processing in frames for sparse approximation.
In Proceedings of the Conference on Information Sciences and
Systems (CISS), Princeton, NJ, March 2008.
Invited paper.
[ .pdf ]
Beyond signal processing applications, frames are also powerful tools
for modeling the sensing and information processing of many biological
and man-made systems that exhibit inherent redundancy. In many cases,
these systems are required to use distributed computational strategies
to analyze and process the sensory information. In this talk, I will
review the use of frames to model distributed sensing systems with a
particular focus on sensory neural systems. In light of the evidence
that many of these systems employ sparse codes, I will describe our
Locally Competitive Algorithms (LCAs) that use a dynamical system to
solve many sparse approximation problems. These LCAs employ a
parallel computational architecture with simple analog components. I
will show numerical simulation results for these systems and describe
their relationship to the many recently-proposed iterative
thresholding algorithms. Our LCA approach also demonstrates potential
advantages in coding time-varying signals (e.g., video) by reflecting
the smooth signal changes in smooth coefficient variations. Finally, I
will highlight some future directions where we hope to impact areas
such as efficient analog signal processing devices, fast discrete
approximation algorithms, and video processing and computer vision in
complex temporal environments.
C.J. Rozell, D.H. Johnson, R.G. Baraniuk, and B.A. Olshausen.
Locally competitive algorithms for sparse approximation.
In Proceedings of the International Conference on Image
Processing (ICIP), pages 169-172, San Antonio, TX, September 2007.
P. Casazza, G. Kutyniok, S. Li, and C.J. Rozell.
Modeling sensor networks with fusion frames.
In Proceedings of SPIE, Wavelets XII at SPIE Optics and
Photonics, volume 6701, pages 67011M-1 - 67011M-11, San Diego, CA, August
2007.
[ .pdf ]
The new notion of fusion frames will be presented in
this article. Fusion frames provide an extensive
framework not only to model sensor networks, but
also to serve as a means to improve robustness or
develop efficient and feasible reconstruction
algorithms. Fusion frames can be regarded as sets of
redundant subspaces each of which contains a
spanning set of local frame vectors, where the
subspaces have to satisfy special overlapping
properties. Main aspects of the theory of fusion
frames will be presented with a particular focus on
the design of sensor networks. New results on the
construction of Parseval fusion frames will also be
discussed.
C.J. Rozell and D.H. Johnson.
Power scheduling for wireless sensor and actuator networks.
In Proceedings of the International Conference on Information
Processing in Sensor Networks (IPSN), pages 470-478, Cambridge, MA, April
2007.
(Acceptance rate of 38/170˜22%.).
[ .pdf ]
We previously presented a model for some wireless sensor and actuator
network (WSAN) applications based on the vector space tools of frame
theory. In this WSAN model there is a weight associated to each
sensor-actuator link denoting the importance of that communication
link to the actuation fidelity. These weights were shown to be useful
in pruning away communication links to reduce the number of active
channels. Inspired by recent work in power scheduling for
decentralized estimation, we investigate the optimal allocation of
system resources for achieving a desired actuation fidelity. In this
scheme, each sensor acquires a noisy observation and sends a message
to a subset of actuators using an MQAM transmission strategy. The
message sent on each sensor-actuator communication link is quantized
with a variable number of bits, with the number of bits optimized to
minimize the total network power consumption subject to a constraint
on the actuation distortion. We show analytically and verify through
simulation that performing this optimal power scheduling can yield
significant power savings over communication strategies that use a
fixed number of bits on each communication link.
S.W. Bishnoi, C.S. Levin, C.J. Rozell, B.R. Johnson, D.H. Johnson, and N.J
Halas.
All-optical nanoscale pH meter: a plasmonic nanodevice with
quantifiable output.
In Proceedings of the Annual Meeting of the IEEE Lasers and
Electro-Optics Society (IEEE LEOS), Montreal, Canada, October 2006.
Invited paper.
C.J. Rozell and D.H. Johnson.
Evaluating local contributions to global performance in wireless
sensor and actuator networks.
Lecture Notes in Computer Science, 4026:1-16, June 2006.
Proceedings of the International Conference on Distributed
Computing in Sensor Systems (DCOSS), San Francisco, CA, June 2006.
[ .pdf ]
Wireless sensor networks are often studied with the goal of removing
information from the network as efficiently as possible. However,
when the application also includes an actuator network, it is
advantageous to determine actions in-network. In such settings,
optimizing the sensor node behavior with respect to sensor information
fidelity does not necessarily translate into optimum behavior in terms
of action fidelity. Inspired by neural systems, we present a model of
a sensor and actuator network based on the vector space tools of
frame theory that applies to applications analogous to reflex
behaviors in biological systems. Our analysis yields bounds on both
absolute and average actuation error that point directly to strategies
for limiting sensor communication based not only on local measurements
but also on a measure of how important each sensor-actuator link is to
the fidelity of the total actuation output.
C.J. Rozell, I.N. Goodman, and D.H. Johnson.
Feature-based information processing with selective attention.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), Toulouse, France, May 2006.
[ .pdf ]
We present a simple but general model for feature-based
information processing with selective attention. We model
feature extraction as projections onto frames of subspaces,
which accounts for redundancies in the representations of
individual features as well as between features. To manage
limited resources, we use feedback attentional signals to
dynamically allocate system resources according to the
observed events. In our model, attention maximizes the
average information retained about all events weighted by
their relative priorities. We illustrate the model with a
simple system under a total bit constraint and discuss how
the organization of the feature extraction affects the
optimal bit allocation.
C.J. Rozell and D.H. Johnson.
Analysis of noise reduction in redundant expansions under distributed
processing requirements.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 185-188, Philadelphia, PA,
March 2005.
[ .pdf ]
We considered signal reconstruction with
redundant expansions under distributed processing in noisy environments.
Redundant expansions have the ability to reduce noise corrupting the
coefficients, but distributed processing schemes will not be able
to take full advantage of the redundancy present. We apply frame
theory and a generalization called “frames of subspaces” to find
conditions when distributed reconstruction suffers no loss in noise
reduction ability, and we bound performance loss in more general cases.
M.A. Lexa, C.J. Rozell, S. Sinanović, and D.H. Johnson.
To cooperate or not to cooperate: Detection strategies in sensor
networks.
In Proceedings of the International Conference on Acoustics,
Speech, and Signal Processing (ICASSP), pages 841-844, Montreal, Canada,
May 2004.
[ .pdf ]
This paper is an initial investigation into the following
question: Can cooperation among sensors in a sensor network improve
detection performance in a simple hypothesis test? We analyze a simple
cooperative system using the Kullback-Leibler (KL) discrimination distance
and a quantity known as the information transfer ratio which is a
ratio of KL distances. We discover that, asymptotically, gain over a
non-cooperative system depends on the conditional KL distance. We conclude
with an illustrative example which demonstrates that cooperation not
only significantly improves performance but can also degrade it.
C.J. Rozell and D. Manolakis.
Matched filter performance for unequal target and background
covariance matrices.
In Proceedings of the SPIE Defense and Security Symposium:
Algorithms and Technologies for Multispectral, Hyperspectral, and
Ultraspectral Imagery X, pages 109-117, Orlando, FL, April 2004.
Detection of military and civilian targets from
airborne platforms using hyperspectral imaging (HSI) sensors is of
great interest. Relative to multispectral sensing, hyperspectral
sensing can increase the detectability of targets by exploiting finer
detail in spectral signatures. A multitude of adaptive detection
algorithms have appeared in the literature or have found their way into
software packages and end-user systems. The most widely known among them
is the linear matched filter. However, despite its popularity, the
fact that the matched filter is used under conditions that deviate
from the implicit optimality assumptions has not been investigated.
M. Simoni, B. Broening, C. Rozell, C. Meek, and G. Wakefield.
A theoretical framework for electro-acoustic music.
In International Computer Music Conference (ICMC), Beijing,
China, 1999.
[ .pdf ]
H.L. Yap and C.J. Rozell.
On the relation between block diagonal matrices and compressive
Toeplitz matrices.
Technical report, Georgia Institute of Technology, School of
Electrical and Computer Engineering, October 2011.
[ .pdf ]
In a typical communications problem, Toeplitz matrices Φ arise when modeling the task of determining an unknown impulse response a from a given probe signal φ. When a is sparse, then whenever Φ formed from the probe signal φ satisfy the Restricted Isometry Property (RIP), a can be robustly recovered from its measurements via l1-minimization. In this paper, we derived the RIP for compressive Toeplitz matrices whose number of rows of the matrices J is much smaller than the number of columns N. We show that J should scale like J ~S2 log(N), where S is the sparsity of the impulse response. While this is marginally worse than the state-of-the-art scaling currently achieved in the literature, the novelty of this work comes from making the relation between the Toeplitz matrix of interest to a block diagonal matrix. The proof of the RIP then follows from using recent results on the concentration of measure inequalities of block diagonal matrices, together with a standard covering-and-counting argument.
C.J. Rozell.
Distributed redundant representations in man-made and biological
sensing systems.
PhD thesis, Rice University, Houston, TX, May 2007.
[ .pdf ]
The ability of a man-made or biological system to understand its
environment is limited by the methods used to process sensory
information. In particular, the data representation is often a
critical component of such systems. Neural systems represent sensory
information using distributed populations of neurons that are highly
redundant. Understanding the role of redundancy in
distributed systems is important both to understanding neural systems
and to efficiently solving many modern signal processing problems.
This thesis makes contributions to understanding redundant
representations in distributed processing systems in three specific
areas. First, we explore the robustness of redundant representations
by generalizing existing results regarding noise-reduction to
Poisson process modulation. Additionally, we characterize how the
noise-reduction ability of redundant representation is weakened when
we enforce a distributed processing constraint on the system.
Second, we explore the task of managing redundancy in the context of
distributed settings through the specific example of wireless sensor
and actuator networks (WSANs). Using a crayfish reflex behavior as a
guide, we develop an analytic WSAN model that implements control laws
in a completely distributed manner. We also develop an algorithm to
optimize the system resource allocation by adjusting the number of
bits used to quantize messages on each sensor-actuator communication
link. This optimal power scheduling yields several orders of
magnitude in power savings over uniform allocation strategies that use
a fixed number of bits on each communication link.
Finally, we explore the flexibility of redundant representations for
sparse approximation. Neuroscience and signal processing both need a
sparse approximation algorithm (i.e., representing a signal with few
non-zero coefficients) that is physically implementable in a parallel
system and produces smooth coefficient time-series for time-varying
signals (e.g., video). We present a class of locally competitive
algorithms (LCAs) that minimize a weighted combination of
mean-squared error and a coefficient cost function. LCAs produce
coefficients with sparsity levels comparable to centralized algorithms
while being more realistic for physical implementation. The resultant
LCA coefficients for video sequences are more regular (i.e., smoother
and more predictable) than the coefficients produced by existing
algorithms.
C.J. Rozell.
Dynamic systems for sparse coding.
Technical report, Walter Karplus Summer Research Grant Final Report,
February 2007.
Despite evidence that neural systems may be employing sparse
stimulus coding, the mechanisms underlying this ability are not
understood. I present a class of neurally plausible locally
competitive algorithms (LCAs) that correspond to a collection of
sparse approximation principles. These systems minimize a combination
of MSE and a coefficient cost function. LCAs use a combination of
thresholding and inhibitory connections to induce local (possibly
one-way) competitions. LCAs demonstrate sparsity levels comparable to
existing sparse coding algorithms while being much more realistic for
neural implementation. Additionally, LCAs coefficients for video
sequences demonstrate inertial properties, making them both
qualitatively and quantitatively more regular (i.e., smoother and more
predictable) than the coefficients produced by greedy algorithms. In
addition to being a first step toward an experimentally testable
hypothesis of biological mechanisms, LCAs may provide new methods for
hardware implementations of sensing systems (e.g., digital cameras)
and new front-end representations for both video coding and computer
vision tasks.
C.J. Rozell.
Analyzing dynamics and stimulus feature dependence in the information
processing of crayfish sustaining fibers.
Master's thesis, Rice University, Houston, TX, May 2002.
[ .pdf ]
The sustaining fiber (SF) stage of the crayfish
visual system converts analog stimulus representations to spike train
signals. A recent theory quantifies a system's information processing
capabilities and relates to statistical signal processing. To analyze
SF responses to light stimuli, we extend a wavelet-based algorithm
for separating analog input signals and spike output waveforms in
composite intracellular recordings. We also present a time-varying
RC circuit model to capture nonstationary membrane noise spectral
characteristics. In our SF analysis, information transfer ratios are
generally on the order of 10-4. The SF information processing
dynamics show transient peaks followed by decay to steady-state values.
A simple theoretical spike generator is analyzed analytically and
shows general dynamic and steady-state properties similar to SFs. The
information transfer ratios increase with spike rate and dynamic
properties are due to direct spike generator dependence on input changes.
This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.This file has been generated by
bibtex2html 1.66
H.L. Yap and C.J. Rozell.
On the relation between block diagonal matrices and compressive
Toeplitz matrices.
Technical report, Georgia Institute of Technology, School of
Electrical and Computer Engineering, October 2011.
[ .pdf ]
In a typical communications problem, Toeplitz matrices Φ arise when modeling the task of determining an unknown impulse response a from a given probe signal φ. When a is sparse, then whenever Φ formed from the probe signal φ satisfy the Restricted Isometry Property (RIP), a can be robustly recovered from its measurements via l1-minimization. In this paper, we derived the RIP for compressive Toeplitz matrices whose number of rows of the matrices J is much smaller than the number of columns N. We show that J should scale like J ~S2 log(N), where S is the sparsity of the impulse response. While this is marginally worse than the state-of-the-art scaling currently achieved in the literature, the novelty of this work comes from making the relation between the Toeplitz matrix of interest to a block diagonal matrix. The proof of the RIP then follows from using recent results on the concentration of measure inequalities of block diagonal matrices, together with a standard covering-and-counting argument.
C.J. Rozell.
Distributed redundant representations in man-made and biological
sensing systems.
PhD thesis, Rice University, Houston, TX, May 2007.
[ .pdf ]
The ability of a man-made or biological system to understand its
environment is limited by the methods used to process sensory
information. In particular, the data representation is often a
critical component of such systems. Neural systems represent sensory
information using distributed populations of neurons that are highly
redundant. Understanding the role of redundancy in
distributed systems is important both to understanding neural systems
and to efficiently solving many modern signal processing problems.
This thesis makes contributions to understanding redundant
representations in distributed processing systems in three specific
areas. First, we explore the robustness of redundant representations
by generalizing existing results regarding noise-reduction to
Poisson process modulation. Additionally, we characterize how the
noise-reduction ability of redundant representation is weakened when
we enforce a distributed processing constraint on the system.
Second, we explore the task of managing redundancy in the context of
distributed settings through the specific example of wireless sensor
and actuator networks (WSANs). Using a crayfish reflex behavior as a
guide, we develop an analytic WSAN model that implements control laws
in a completely distributed manner. We also develop an algorithm to
optimize the system resource allocation by adjusting the number of
bits used to quantize messages on each sensor-actuator communication
link. This optimal power scheduling yields several orders of
magnitude in power savings over uniform allocation strategies that use
a fixed number of bits on each communication link.
Finally, we explore the flexibility of redundant representations for
sparse approximation. Neuroscience and signal processing both need a
sparse approximation algorithm (i.e., representing a signal with few
non-zero coefficients) that is physically implementable in a parallel
system and produces smooth coefficient time-series for time-varying
signals (e.g., video). We present a class of locally competitive
algorithms (LCAs) that minimize a weighted combination of
mean-squared error and a coefficient cost function. LCAs produce
coefficients with sparsity levels comparable to centralized algorithms
while being more realistic for physical implementation. The resultant
LCA coefficients for video sequences are more regular (i.e., smoother
and more predictable) than the coefficients produced by existing
algorithms.
C.J. Rozell.
Dynamic systems for sparse coding.
Technical report, Walter Karplus Summer Research Grant Final Report,
February 2007.
Despite evidence that neural systems may be employing sparse
stimulus coding, the mechanisms underlying this ability are not
understood. I present a class of neurally plausible locally
competitive algorithms (LCAs) that correspond to a collection of
sparse approximation principles. These systems minimize a combination
of MSE and a coefficient cost function. LCAs use a combination of
thresholding and inhibitory connections to induce local (possibly
one-way) competitions. LCAs demonstrate sparsity levels comparable to
existing sparse coding algorithms while being much more realistic for
neural implementation. Additionally, LCAs coefficients for video
sequences demonstrate inertial properties, making them both
qualitatively and quantitatively more regular (i.e., smoother and more
predictable) than the coefficients produced by greedy algorithms. In
addition to being a first step toward an experimentally testable
hypothesis of biological mechanisms, LCAs may provide new methods for
hardware implementations of sensing systems (e.g., digital cameras)
and new front-end representations for both video coding and computer
vision tasks.
C.J. Rozell.
Analyzing dynamics and stimulus feature dependence in the information
processing of crayfish sustaining fibers.
Master's thesis, Rice University, Houston, TX, May 2002.
[ .pdf ]
The sustaining fiber (SF) stage of the crayfish
visual system converts analog stimulus representations to spike train
signals. A recent theory quantifies a system's information processing
capabilities and relates to statistical signal processing. To analyze
SF responses to light stimuli, we extend a wavelet-based algorithm
for separating analog input signals and spike output waveforms in
composite intracellular recordings. We also present a time-varying
RC circuit model to capture nonstationary membrane noise spectral
characteristics. In our SF analysis, information transfer ratios are
generally on the order of 10-4. The SF information processing
dynamics show transient peaks followed by decay to steady-state values.
A simple theoretical spike generator is analyzed analytically and
shows general dynamic and steady-state properties similar to SFs. The
information transfer ratios increase with spike rate and dynamic
properties are due to direct spike generator dependence on input changes.
This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.This file has been generated by
bibtex2html 1.66
H.L. Yap and C.J. Rozell.
On the relation between block diagonal matrices and compressive
Toeplitz matrices.
Technical report, Georgia Institute of Technology, School of
Electrical and Computer Engineering, October 2011.
[ .pdf ]
In a typical communications problem, Toeplitz matrices Φ arise when modeling the task of determining an unknown impulse response a from a given probe signal φ. When a is sparse, then whenever Φ formed from the probe signal φ satisfy the Restricted Isometry Property (RIP), a can be robustly recovered from its measurements via l1-minimization. In this paper, we derived the RIP for compressive Toeplitz matrices whose number of rows of the matrices J is much smaller than the number of columns N. We show that J should scale like J ~S2 log(N), where S is the sparsity of the impulse response. While this is marginally worse than the state-of-the-art scaling currently achieved in the literature, the novelty of this work comes from making the relation between the Toeplitz matrix of interest to a block diagonal matrix. The proof of the RIP then follows from using recent results on the concentration of measure inequalities of block diagonal matrices, together with a standard covering-and-counting argument.
C.J. Rozell.
Distributed redundant representations in man-made and biological
sensing systems.
PhD thesis, Rice University, Houston, TX, May 2007.
[ .pdf ]
The ability of a man-made or biological system to understand its
environment is limited by the methods used to process sensory
information. In particular, the data representation is often a
critical component of such systems. Neural systems represent sensory
information using distributed populations of neurons that are highly
redundant. Understanding the role of redundancy in
distributed systems is important both to understanding neural systems
and to efficiently solving many modern signal processing problems.
This thesis makes contributions to understanding redundant
representations in distributed processing systems in three specific
areas. First, we explore the robustness of redundant representations
by generalizing existing results regarding noise-reduction to
Poisson process modulation. Additionally, we characterize how the
noise-reduction ability of redundant representation is weakened when
we enforce a distributed processing constraint on the system.
Second, we explore the task of managing redundancy in the context of
distributed settings through the specific example of wireless sensor
and actuator networks (WSANs). Using a crayfish reflex behavior as a
guide, we develop an analytic WSAN model that implements control laws
in a completely distributed manner. We also develop an algorithm to
optimize the system resource allocation by adjusting the number of
bits used to quantize messages on each sensor-actuator communication
link. This optimal power scheduling yields several orders of
magnitude in power savings over uniform allocation strategies that use
a fixed number of bits on each communication link.
Finally, we explore the flexibility of redundant representations for
sparse approximation. Neuroscience and signal processing both need a
sparse approximation algorithm (i.e., representing a signal with few
non-zero coefficients) that is physically implementable in a parallel
system and produces smooth coefficient time-series for time-varying
signals (e.g., video). We present a class of locally competitive
algorithms (LCAs) that minimize a weighted combination of
mean-squared error and a coefficient cost function. LCAs produce
coefficients with sparsity levels comparable to centralized algorithms
while being more realistic for physical implementation. The resultant
LCA coefficients for video sequences are more regular (i.e., smoother
and more predictable) than the coefficients produced by existing
algorithms.
C.J. Rozell.
Dynamic systems for sparse coding.
Technical report, Walter Karplus Summer Research Grant Final Report,
February 2007.
Despite evidence that neural systems may be employing sparse
stimulus coding, the mechanisms underlying this ability are not
understood. I present a class of neurally plausible locally
competitive algorithms (LCAs) that correspond to a collection of
sparse approximation principles. These systems minimize a combination
of MSE and a coefficient cost function. LCAs use a combination of
thresholding and inhibitory connections to induce local (possibly
one-way) competitions. LCAs demonstrate sparsity levels comparable to
existing sparse coding algorithms while being much more realistic for
neural implementation. Additionally, LCAs coefficients for video
sequences demonstrate inertial properties, making them both
qualitatively and quantitatively more regular (i.e., smoother and more
predictable) than the coefficients produced by greedy algorithms. In
addition to being a first step toward an experimentally testable
hypothesis of biological mechanisms, LCAs may provide new methods for
hardware implementations of sensing systems (e.g., digital cameras)
and new front-end representations for both video coding and computer
vision tasks.
C.J. Rozell.
Analyzing dynamics and stimulus feature dependence in the information
processing of crayfish sustaining fibers.
Master's thesis, Rice University, Houston, TX, May 2002.
[ .pdf ]
The sustaining fiber (SF) stage of the crayfish
visual system converts analog stimulus representations to spike train
signals. A recent theory quantifies a system's information processing
capabilities and relates to statistical signal processing. To analyze
SF responses to light stimuli, we extend a wavelet-based algorithm
for separating analog input signals and spike output waveforms in
composite intracellular recordings. We also present a time-varying
RC circuit model to capture nonstationary membrane noise spectral
characteristics. In our SF analysis, information transfer ratios are
generally on the order of 10-4. The SF information processing
dynamics show transient peaks followed by decay to steady-state values.
A simple theoretical spike generator is analyzed analytically and
shows general dynamic and steady-state properties similar to SFs. The
information transfer ratios increase with spike rate and dynamic
properties are due to direct spike generator dependence on input changes.
This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.This file has been generated by
bibtex2html 1.66
