‘Types of Activation Function ‘The activation function, denoted by ov), defines the output of a neuron in terms of the induced local field v. Here we identify three basic types of activation functions: 1. Threshold Function. For this type of activation function, described in Fig. 1.8a, we have 1 ifv=o oo ={j wa (18) In engineering literature, this form of a threshold function is commonly referred to as a Heaviside function. Correspondingly, the output of neuron k employing such a thresh- old function is expressed as _ fi ify2o Ja fo ify <0 > where 2, is the induced local field of the neuron; that is, = Ywyx; + be (1.10) a troy ia Filings Ge [ “= Activation no ee sree ems actin as ronlnea motoneuron . t va 7 7 7 7 ee @12; 0 | 08! Increasing FIGURE 1.8 (a) Threshold function. (b) Piecewise-linear function. (¢) Sigmoid function © for varying slope parameter a. Such a neuron is referred to in the literature as the McCulloch-Pitts model, in recogni- tion of the pioneering work done by McCulloch and Pitts (1943). In this model, the ‘output of a neuron takes on the value of 1 if the induced local field of that neuron is, nonnegative, and 0 otherwise. This statement describes the all-or-none property of the McCulloch Pitts model. 2. Piecewise-Linear Function. For the piecewise-linear function described in Fig. 1.8b we have 1, vet) ev) = 40, tout (ayy 0, ve-t where the amplification factor inside the linear region of operation is assumed to be unity. This form of an activation function may be viewed as an approximation to a non- \ear amplifier. The following two situations may be viewed as special forms of the piecewise-linear function: * A linear combiner arises if the linear region of operation is maintained without running into saturation. + The piecewise-linear function reduces to a threshold function if the amplification factor of the linear region is made infinitely large.3. Sigmoid Function. The sigmoid function, whose graph is s-shaped, is by far the ‘most common form of activation function used in the construction of artifical neural networks, It is defined as a strictly increasing function that exhibits a graceful balance between linear and nonlinear behavior. An example of the sigmoid function is the logistic function, defined by 1 T+ exp(-av) 90) = (1.12) where a is the slope parameter of the sigmoid function. By varying the parameter a, we obtain sigmoid functions of different slopes, as illustrated in Fig. 1.8c. Infact, the slope at the origin equals a/4. In the limit, as the slope parameter approaches infinity, the sig- ‘moid function becomes simply a threshold function. Whereas a threshold function assumes the value of 0 or 1, a sigmoid function assumes a continuous range of values from 0 to 1. Note also that the sigmoid function is differentiable, whereas the threshold function is not. (Differentiability is an important feature of neural network theory, as described in Chapter 4.) ‘The activation functions defined in Eqs. (1.8), (1.11), and (1.12) range from 0 to +1, It is sometimes desirable to have the activation function range from ~1 to +1, in which case the activation function assumes an antisymmetric form with respect to the origin; that is, the activation function is an odd function of the induced local field. Specifically, the threshold function of Eq. (1.8) is now defined as, 1 ifv>o e)=4 0 ifv=0 (1.13) -1 ifv<0 which is commonly referred to as the signuom function, For the corresponding form of a sigmoid function we may use the hyperbolic tangent function, defined by 9(v) = tanh(o) (14) Allowing an activation function of the sigmoid type to assume negative values as pre- scribed by Eq, (1.14) has analytic benefits (as shown in Chapter 4),