Linearity vs. Nonlinearity
Don't be crooked! Remember what I told you about going straight?
---------------------------------------------------------------------------
------------------------------------------------
In a linear context, if I were to buy ten pounds of grapes instead of five, I would pay twice as much. If I triple the amount, I'd triple the cost. Linear means proportional. The unit price is unaffected by the amount of the purchase. The percentage remains the same. There is no favoritism in the price -- no memory.
Mathematically, this relationship is expressed by the simple formula: y = ax, where 'a' is a parameter, which can be said to confine the environmental conditions. For any given a, the equation would have a straight-line graphical form. Increasing the value of x increases the value of y, proportionately, by the factor a.
-----------------------------------------------
In a nonlinear context, reaction to inputs is not proportional to the inputs. For instance, the price you pay for grapes would likely be less per unit if you were to buy in large amounts. That is, the amount you buy makes a difference -- the results are not blind to the quantity of the purchase. The purchase favors a larger amount by granting a lower cost. In this sense there is memory; for the amounts to be significant, they must be remembered as having special value.
But there are different things with different values to "remember." So there is no single form. The kind of non-linearity to be used in practice depends on the environmental relationships. The value of y might increase as the square of the value of x. In this case, doubling x would quadruple y. So, if x were to go from 2 to 4, y would jump from 4a to 16a. This is obviously not a straight-line function. You might say the function is biased toward higher values of x. The system "remembers" that higher values of x are more significant.
In another situation, x itself could also depend on y, just as y depends on x. One person could affect another in some way and also be affected by the other, very likely in a different way. In this case, two equations that are linear when considered independently, combine to form a nonlinear system of equations. The value of x feeds back to affect y, and y feeds back to affect x. Feedback is an important ingredient of neural networks.
---------------------------------------