INFORMATION AND COMMUNICATION THEORY
Unit-1: Probability Theory
Unit-2: Stochastic Processes
Unit-3: Estimation & Hypothesis Testing
Unit-4: Information Theory
Unit-5: Statistical Modeling of Noise
Module-3
• Stochastic Processes
• Statistical Modeling of Noise
Stochastic (Random) Process
Random variable is a rule to assign a number to every outcome ξ of an
experiment .
A stochastic (Random) process is a rule to assign a function to every outcome ξ of
an experiment .
Experiment: Tossing of a coin Experiment: Tossing of a coin
= Number of Heads = Number of Heads
If outcome If outcome
If outcome If outcome
RV assigns a real-number to RP assigns a real-function of time ‘t’ to
each outcome of the experiment each outcome of the experiment
Represented by or Represented by or
Stochastic (Random) Process
Stochastic process is represented by . It has following interpretations –
• If are variables: It is a family (or ensemble) of functions .
• If is a variable but is fixed: It is a single time-function (or a sample of the given
process).
• If is a variable but is fixed: It is the random variable equal to the state of given
process at time .
• If both are fixed: It is a real-number.
CDF of Random Process
For a specific , is random variable with distribution
This function depend on time and is equal to the probability of the event
consisting of all outcomes , such that, at specific time , the samples of of the
given process does not exceed.
is called first-order distribution of the RP
is the first-order density of the process
For given and , and are two RVs with distribution
is called as second-order distribution of the RP
Mean Function and Correlation Function
The mean function gives an idea about how the random process behaves on average as time evolves. It
is function of time and also called as ENSEMBLE average of . At a specific , is random variable and its
mean will be
At two time-instants and , and are random variables. The correlation between them is defined as –
• called as expected (average) power in at time t. For a WSS process, the expected power is
not a function of time.
• i.e. is an even function.
•
Proof: by Cauchy-Schwarz inequality
Correlation and Covariance Functions
The autocovariance function, is defined for a random process
If
Example on CDF, Mean, Correlation and Covariance function
Example on CDF, Mean, Correlation and Covariance function
Stationary Process
Stationary Processes
A random process is stationary if its statistical properties do not change by time
i.e. for a stationary process, and have the same probability distributions.
Also for a stationary process, the joint distribution of are the same as the joint
distribution of .
In practice, it is desirable that a random process is stationary because its analysis
is usually simpler as the probabilistic properties do not change by time.
Stationary Processes
Weak-Sense Stationary Processes
A random process is called weak-sense stationary or wide-sense stationary (WSS) if
its mean function and its correlation function do not change by shifts in time. More
precisely, is WSS if, for all and ,
i.e.
• mean function is not a function of time
• correlation function is only a function of and not individually.
Weak-Sense Stationary Processes
Cyclostationary Processes
Some practical random processes have a periodic structure. That is, the statistical
properties are repeated every units of time (e.g., every seconds). In other words,
the random variables have the same joint CDF as the random variables . Such
random variables are called cyclostationary.
A continuous-time random process is weak-sense cyclostationary or wide-sense
cyclostationary if there exists a positive real number such that
Example: . which is a periodic signal with period . Therefore, is a cyclostationary
random process with period .
Power Spectral Density
Consider a WSS random process with autocorrelation function . Then the Power
Spectral Density (PSD) of can be defined as the Fourier transform of .
Thus
If is a real-valued random process, then is an even, real-valued function of . From
the properties of the Fourier transform, it can be concluded that is also real-valued
and an even function of .
and
Power Spectral Density ∞
𝑅 𝑋 ( 𝜏 )= ∫ 𝑆 𝑋 ( 𝑓 ) 𝑒
𝑗 2𝜋 𝑓 𝜏
𝑑𝑓
For , the expected (or average power) in −∞
i.e. the expected power in can be obtained by integrating the PSD of .
Therefore, power spectrum of a stochastic process can easily be obtained by its power spectral
density which in turn can be obtained by Fourier transform of autocorrelation function of that
random process.
Cross Spectral Density:
For two jointly WSS random processes and , the cross spectral density can be
defined as the Fourier transform of the cross-correlation function
Systems with Stochastic Inputs
If each sample of a stochastic process is assigned some function according
to some rule , then will be another stochastic process, given as
System is deterministic if it operates only on variable treating as parameter
i.e.
If and of are identical and of are also identical.
System is stochastic if it operates on both variables and i.e.
of are identical in at .
Linear Time-Invariant (LTI) Systems
Linear time-invariant systems (LTI systems) are a class of systems used
in signals and systems that are both linear and time-invariant.
Linear systems are systems whose outputs for a linear combination of inputs
are the same as a linear combination of individual responses to those inputs.
In other words, for a system over time , composed of signals and with
outputs and ,
Time-invariant systems are systems where the output does not depend
on when an input was applied i.e.
Where is some delay.
Linear Time-Invariant (LTI) Systems
.
Mean of output of LTI Systems
Consider an LTI system with impulse response . Let be a WSS random
process. If is the input of the system, then the output, , is also a random
process.
So
Cross-correlation Function of output of LTI Systems
where
Einstein-Wiener-Khintchine relation
Properties of Power Spectral Density
Property-1: The zero-frequency value of PSD of a stationary process equals
the total area under the graph of auto-correlation function
Property-2: The mean-square value of stationary process is equal to the total
area under the graph of PSD
Property-3: The PSD of a stationary process is always non-negative
Property-4: The PSD of a real-valued random process is an even function of
frequency
Properties of Power Spectral Density
Property-5: The PSD, appropriately normalized, has the properties usually
Defining
associated with a probability density function.
For this function
• as
Reference
• Athanasios Papoulis, “Probability, Random Variables, and Stochastic
Processes,” 3rd edition, McGraw Hill Publication.
• https://www.probabilitycourse.com
