![[Mensa]](http://www.ingber.com/mensa.jpg)
"This above all; to thine own self be true."
William Shakespeare (Polonius' advice to his son Laertes), "Hamlet" (1601).
"... great life, if you don't weaken"
Irving Klock, Farmer, Herkimer County, NY (1956)
"Thinking about thinking can make you crazy."
Richard Feynman, Caltech, communication to Lester Ingber (1970).
Lester Ingber Research (LIR) develops projects in areas of
expertise documented in the ingber.com InterNet archive. Terms of use,
downloading policies, and consulting/contracting are discussed in the file
ingber_terms.html
A one-minute video introduction is in MPEG1 or AVI formats:
ingber_projects_video.html
Jul 06
This project is described in a report:
ABSTRACT: Ideas by Statistical Mechanics (ISM) is a generic program to model evolution and propagation of ideas/patterns throughout populations subjected to endogenous and exogenous interactions. The program is based on the author's work in Statistical Mechanics of Neocortical Interactions (SMNI), and uses the author's Adaptive Simulated Annealing (ASA) code for optimizations of training sets, as well as for importance-sampling to apply the author's copula financial risk-management codes, Trading in Risk Dimensions (TRD), for assessments of risk and uncertainty. This product can be used for decision support for projects ranging from diplomatic, information, military, and economic (DIME) factors of propagation/evolution of ideas, to commercial sales, trading indicators across sectors of financial markets, advertising and political campaigns, etc.
It seems appropriate to base an approach for propagation of ideas on the only system so far demonstrated to develop and nurture ideas, i.e., the neocortical brain. A statistical mechanical model of neocortical interactions, developed by the author and tested successfully in describing short-term memory and EEG indicators, is the proposed model. ISM develops subsets of macrocolumnar activity of multivariate stochastic descriptions of defined populations, with macrocolumns defined by their local parameters within specific regions and with parameterized endogenous inter-regional and exogenous external connectivities. Parameters with a given subset of macrocolumns will be fit using ASA to patterns representing ideas. Parameters of external and inter-regional interactions will be determined that promote or inhibit the spread of these ideas. Tools of financial risk management, developed by the author to process correlated multivariate systems with differing non-Gaussian distributions using modern copula analysis, importance-sampled using ASA, will enable bona fide correlations and uncertainties of success and failure to be calculated. Marginal distributions will be evolved to determine their expected duration and stability using algorithms developed by the author, i.e., PATHTREE and PATHINT codes.
Dec 03
Oct 04
Jan 06
Apr 06
Previous work, mostly published, developed two-shell recursive trading systems. An inner-shell of Canonical Momenta Indicators (CMI) is adaptively fit to incoming market data. A parameterized trading-rule outer-shell uses the global optimization code Adaptive Simulated Annealing (ASA) to fit the trading system to historical data. A simple fitting algorithm, usually not requiring ASA, is used for the inner-shell fit.
An additional risk-management middle-shell has been added to create a three-shell recursive optimization/sampling/fitting algorithm. Portfolio-level distributions of copula-transformed multivariate distributions (with constituent markets possessing different marginal distributions in returns space) are generated by Monte Carlo samplings. ASA is used to importance-sample weightings of these markets. There are many publications in the academic and commercial literature on similar treatments of markets. The Buddha is in the details :).
TRD processes Training and Testing trading systems on historical data, and consistently interacts with RealTime trading platforms -- all at minute resolutions. Faster or slower resolutions can be developed using the present structure of TRD. The code is written in vanilla C, and runs across platforms such as XP/Cygwin, SPARC/Solaris, i386/FreeBSD, i386/NetBSD, etc. TRD can be run as an independent executable or called as a DLL.
To illustrate how TRD can robustly and flexibly interact with various trading platforms, I have developed a working interface with TradeStation and have outlined an interface with Fidelity's Wealth-Lab.
The LIR basic concepts, architecture, and TRD code described above can be used to further develop LIR's published approaches to projects in other disciplines, as well as to generate other projects.
These concepts can be applied to developing tools for decision-makers of companies and government agencies, assembling functions, departments, processes, etc., tailored to specific client requirements, into an overall "portfolio" from which top-level measures of performance are developed with associated measures of risk, and with audit trails back to the member constituents. For example, marginal distributions can be formulated based on many kinds of sets of data. A report describes how options (in the sense of financial options) and risk management can be developed for project schedules:
Adaptive Simulated Annealing (ASA) and path-integral techniques, e.g., PATHINT and PATHTREE, have been published, demonstrating their utility in statistical mechanical studies in finance, neuroscience, combat analyses, neuroscience, and other selected nonlinear multivariate systems. PATHTREE has been used extensively to price financial options.
PATHTREE can be be generalized, from folding forward in time a wide class of nonlinear stochastic 1-dimensional distributions, to n-dimensional distributions in an n-PATHTREE code. PATHINT becomes extremely computationally intensive with just 3 variables, but n-PATHTREE likely can be much quicker with the same extreme accuracy for higher dimensional systems. A matrix formulation of binomial trees could be the focus for developing n-PATHTREE. This algorithm can be used to fit the shape of distributions to data, providing a robust bottom-up approach to "curve-fitting" systematics of empirical data.
These tools also are being applied to price complex projects as financial options with alternative schedules and strategies. PATHTREE processes real-world options, including nonlinear distributions and time-dependent starting and stopping of sub-projects, with parameters of shapes of distributions fit using ASA to optimize cost and duration of sub-projects.
High-quality EEG data has been used to perform ASA optimization of "canonical momenta" indicators (CMI). This work demonstrates how the theory of statistical mechanics of neocortical interactions (SMNI) can describe individuals' macroscopic brain function as measured by raw EEG.
The first paper giving a detailed calculation of CMI in EEG was
ABSTRACT: A series of papers has developed a statistical mechanics of neocortical interactions (SMNI), deriving aggregate behavior of experimentally observed columns of neurons from statistical electrical-chemical properties of synaptic interactions. While not useful to yield insights at the single neuron level, SMNI has demonstrated its capability in describing large-scale properties of short-term memory and electroencephalographic (EEG) systematics. The necessity of including nonlinear and stochastic structures in this development has been stressed. Sets of EEG and evoked potential data were fit, collected to investigate genetic predispositions to alcoholism and to extract brain "signatures" of short-term memory. Adaptive Simulated Annealing (ASA), a global optimization algorithm, was used to perform maximum likelihood fits of Lagrangians defined by path integrals of multivariate conditional probabilities. Canonical momenta indicators (CMI) are thereby derived for individual's EEG data. The CMI give better signal recognition than the raw data, and can be used to advantage as correlates of behavioral states. These results give strong quantitative support for an accurate intuitive picture, portraying neocortical interactions as having common algebraic or physics mechanisms that scale across quite disparate spatial scales and functional or behavioral phenomena, i.e., describing interactions among neurons, columns of neurons, and regional masses of neurons.
A follow-up study, including testing the CMI on out-of-sample data, is in
Additional results (tables of ASA-fitted parameters and 60 files containing 240 PostScript graphs) are contained in smni97_eeg_cmi.tar.gz
Some background and results in lecture-plate form are given in
Dec 06
Some of the algorithms used in the ISM project (above) are used in
An approach to developing a quasi-automated trading system, based on a theory of a statistical mechanics of financial markets (SMFM) has been developed in several papers, e.g.,
ABSTRACT: A paradigm of statistical mechanics of financial markets (SMFM) is fit to multivariate financial markets using Adaptive Simulated Annealing (ASA), a global optimization algorithm, to perform maximum likelihood fits of Lagrangians defined by path integrals of multivariate conditional probabilities. Canonical momenta are thereby derived and used as technical indicators in a recursive ASA optimization process to tune trading rules. These trading rules are then used on out-of-sample data, to demonstrate that they can profit from the SMFM model, to illustrate that these markets are likely not efficient. This methodology can be extended to other systems, e.g., electroencephalography. This approach to complex systems emphasizes the utility of blending an intuitive and powerful mathematical-physics formalism to generate indicators which are used by AI-type rule-based models of management.
The calculations contained in
A brief and less technical discussion of this approach and of ASA is given in
A brief discussion and motivation for work in progress, further developing SMFM, is given in
Application of SMFM to developing volatility of volatility in the context of Eurodollar options is given in
ABSTRACT: The Black-Scholes theory of option pricing has been considered for many years as an important but very approximate zeroth-order description of actual market behavior. We generalize the functional form of the diffusion of these systems and also consider multi-factor models including stochastic volatility. We use a previous development of a statistical mechanics of financial markets to model these issues. Daily Eurodollar futures prices and implied volatilities are fit to determine exponents of functional behavior of diffusions using methods of global optimization, Adaptive Simulated Annealing (ASA), to generate tight fits across moving time windows of Eurodollar contracts. These short-time fitted distributions are then developed into long-time distributions using a robust non-Monte Carlo path-integral algorithm, PATHINT, to generate prices and derivatives commonly used by option traders. The results of our study show that there is only a very small change in at-the money option prices for different probability distributions, both for the one-factor and two-factor models. There still are significant differences in risk parameters, partial derivatives, using more sophisticated models, especially for out-of-the-money options.
A very quick and robust algorithm motivated by PATHINT, PATHTREE, has been developed:
ABSTRACT: Motivated by path-integral numerical solutions of diffusion processes, PATHINT, we present a new tree algorithm, PATHTREE, which permits extremely fast accurate computation of probability distributions of a large class of general nonlinear diffusion processes.
Some background and results in lecture-plate form are given in
A theory of statistical mechanics of combat (SMC) is given in
ABSTRACT: The context of nonlinear combat calls for more sophisticated measures of effectiveness. We present a set of tools that can be used as such supplemental indicators, based on stochastic nonlinear multivariate modeling used to benchmark Janus simulation to exercise data from the U.S. Army National Training Center (NTC). As a prototype study, a strong global optimization tool, adaptive simulated annealing (ASA), is used to explicitly fit Janus data, deriving coefficients of relative measures of effectiveness, and developing a sound intuitive graphical decision aid, canonical momentum indicators (CMI), faithful to the sophisticated algebraic model. We argue that these tools will become increasingly important to aid simulation studies of the importance of maneuver in combat in the 21st century.
Some background and results in lecture-plate form are given in
21 Oct 95
Here is an edited reply to a query on "canonical momenta," a common feature of the above projects, that may be useful to some readers.
: Just out of curiosity. Could you briefly define "canonical momenta
: indicator?" Very few people understand the definition. If that's
: too much time, please tell me where I could find the information.
Yes, it does require some advanced physics to articulate in detail. When dealing with truly nonlinear multivariate systems, the details go beyond a normal PhD physics education, using some advanced calculus developed in the late '70's.
That said, let me try to give a thumbnail sketch:
A stochastic differential equation (sde) like
This sde can be written as a conditional short-time distribution
The momentum is
In this simple one-dim example, g is just the standard deviation, but in more than one dim, g**2 becomes the covariance matrix. It turns out that this is the inverse-metric of the space as well, and it enters into the calculation of the "canonical momenta." Perhaps the easiest way to see this is to look at the third mathematically equivalent representation, the Fokker-Planck partial differential equation (pde) for p:
A very simple but useful text on the physical relevance of such sde and pdf across many physical and biological systems is
There is a nice chapter on "The Principle of Least Action," illustrating how "F = ma" is derived from the Lagrangian in Volume III, Chapter 19 of
A book that shows all the gory glory of the additional complications that must be dealt with when multivariate nonlinear systems are considered is
My work in neuroscience since the late '70's, e.g., in the smni... papers in my archive, first took advantage of these new mathematical physics developments in multivariate nonequilibrium nonlinear statistical mechanics. I then applied these techniques to markets, nuclear physics, and combat analyses, e.g., in my markets..., nuclear..., and combat... papers in my archive, which give more details and references.
I developed this formalism into a practical maximum likelihood numerical tool for fitting parameters in these kinds of systems with the use of VFSR/ASA, e.g., illustrated with the use of some of the asa... papers in my archive. The ASA code is in
Some background and results in lecture-plate form are given in
The long-time evolution of such multivariate systems is accomplished with the use of algorithms such as PATHINT and PATHTREE, e.g., illustrated in the path... papers in my archive. This is required, for example, to calculate many kinds of financial instruments, e.g., bond prices, options, derivatives, etc. The famous Black-Scholes model for options pricing is such an example of a one-variable distribution. People have developed numerical algorithms for each representation, i.e., for the SDE, PDE, and the Lagrangian probability representations. Examples of the use of PATHINT for options pricing are given in http://www.ingber.com/markets00_exp.pdf and http://www.ingber.com/markets00_highres.pdf and the use of PATHTREE is given in markets01_pathtree.pdf as outlined above.
Some background and results in lecture-plate form are given in
In addition to canonical momenta used as a natural coordinate system to study dynamically evolving multivariate systems, their value has several explicit aspects as used in a code I developed, TRD. With respect to my markets96_trading.pdf and markets96_momenta.pdf papers, in addition to canonical momenta being accurate as well as intuitive measures of "flows" to and from evolving steady states, I have made the points:
These ideas were implemented in a realtime tick-resolution trading system, as described in markets01_optim_trading.pdf
Similarly, in EEG work, it is possible to monitor a subset of electrode activity while incorporating information from additional electrodes.
Papers markets84_statmech.pdf, markets91_interest.pdf and smni91_eeg.pdf contain an Appendix giving a compact derivation of the path-integral Lagrangian representation equivalent to the Langevin rate-equation and Fokker-Planck/Schroedinger-type representations for multivariate systems with nonlinear drifts and diffusions.
My experience with nonlinear systems, based on my own research into selected topics in selected disciplines as well as my interactions with many other experts and nonlinear systems, e.g., through my support of ASA, is that nonlinear systems are typically non-typical. It typically requires quite a bit of work understanding the nature of each each system before applying or evolving models and techniques borrowed from other disciplines.
I hope this helps.
Lester
With just dimension, D = 1, a formula for the probability of having any value x(t) at time t+dt given the value of x(t+dt) at time t would be:
If we are dealing with more than one dimension, D, then G become a matrix and f become a vector, e.g., for x and y:
I have developed the above formulation into a numerical procedure for nonlinear multivariate systems that otherwise would be described as
If [G_i_k] is the matrix of the coefficients in the differential rate equations, and [G_ij] is the covariance matrix (the inverse of the metric matrix for this space) in the Lagrangian discussed above,
As the number of dimensions increases, so do the number of parameters to be fit. As the functions F and G become more nonlinear, as they will with increasing D, and as there definitely is "noise" in the data, it becomes increasingly important to use some powerful global optimization code like ASA to fit the data, else the coefficients will just be modeling the "noise."
An additional hard problem arises in forecasting P for times greater than dt in the future. You have to do the "path integral," i.e., integrals over all the x's for each of the intermediate times:
Only if very simple forms for F and G are selected, can these integrals be performed to give a closed expression; otherwise they must be performed numerically. PATHINT and PATHTREE do this for _some_ classes of integrals. However, even for dimension D = 2 the CPU time is large, and for D = 3 or 4, this might be huge. If simple forms are taken for the F's and G's, all the global optimization fits still must be performed, but perhaps you can get by without using something like PATHINT.
A paper on the general utility of this approach to modeling complex systems is given in
ABSTRACT: A modern calculus of multivariate nonlinear multiplicative Gaussian-Markovian systems provides models of many complex systems faithful to their nature, e.g., by not prematurely applying quasi-linear approximations for the sole purpose of easing analysis. To handle these complex algebraic constructs, sophisticated numerical tools have been developed, e.g., methods of adaptive simulated annealing (ASA) global optimization and of path integration (PATHINT). In-depth application to three quite different complex systems have yielded some insights into the benefits to be obtained by application of these algorithms and tools, in statistical mechanical descriptions of neocortex (short-term memory and electroencephalography), financial markets (interest-rate and trading models), and combat analysis (baselining simulations to exercise data).
Every worthwhile project I've undertaken -- from Nuclear Physics to Statistical Mechanics (SM) projects to Karate -- each took some years of study to understand a basic set of problems in a given discipline. It just takes a lot of time to get familiar with the current issues of a discipline as viewed by experts, to understand the most important unsolved or poorly solved issues, and to see if new approaches can be applied to create better solutions, without being distracted by focussing on easier different problems that do not address the important issues. It is to be expected that a newcomer to any discipline should be expected to do the "homework" required. I addressed real problems requiring such studies, not only to appreciate the core problems at an expert level, but to forge the necessary tools to solve these problems. In the SM studies, this required creating such tools as ASA, PATHINT and PATHTREE, and many other lesser-known algorithms.
Some tools had to be developed to accommodate particular constraints. Similar to optimizing some complex systems, some projects require constraints additional to simply developing analytic solutions to scientific problems, e.g., including an institution's focus, capabilities of team members, ability of decision makers to work with results, etc. These projects can be quite interesting and challenging, requiring blending mathematical science with social or personnel objectives, forming a larger system to be considered. In such cases, once a reasonably solid intuitive understanding is reached for an approach, analysis should strive to not waiver from this intuition, but to be faithful to the original understanding. Of course, often further analysis uncovers new aspects of a system demanding attention and modification, often requiring problems and solutions to converge only at late stages of solution. This is different from simply giving into a too common temptation for analysts to bend their analysis to suit the ease of solution rather than to deliver projects that solve the original problem.
When my Statistical Mechanics of Neocortical Interactions (SMNI) approach first appeared circa 1978, it was not hard to understand why many people in neuroscience trained either in medicine or abstract mathematics could not readily understand this mathematical-physics approach to the neocortex. Especially at that time, neural networks were much simpler to understand, albeit conveniently glossing over issues of relevance to real neocortex. However, even supporters of this work with extremely solid scientific credentials did not stop a few other reviewers from attempting to cut funding and publication of this work. One early reviewer, a well-known neurologist claimed that the math was fabricated as a smoke screen to cover the results claimed by the algebraic and numerical calculations. Since then, SMNI papers have successfully applied this math-physics to numerically detail properties of EEG and short-term memory.
When my Statistical Mechanics of Financial Markets (SMFM) approach first appeared circa 1982, the paper finally published in 1984 was delayed a couple of years. For example, an editor of a premier eonomics journal agreed with a reviewer (who stated he had a graduate-student physics background) that such math as used in that paper did not exist and could not be correct. Since then, this math-physics has successfully been applied to derivatives, trading systems and risk-management of portfolios. Success in trading firms I've worked with, albeit a profitable enterprise, most certainly is not documentation of validity of this work, and I have managed to publish some of this work to get genuine peer review.
When my Statistical Mechanics of Combat (SMC) approach first appeared circa 1987, in the context of studies of studies on large-scale government simulations, several operations-research/statistician government analysts stated a similar complaint that they could not find any such math in their previous math or physics textbooks. One OR professor actually sent out letters to Department Heads claiming I was promoting bogus studies. They claimed "path integrals" were only a theoretical abstraction without any foundation for application to anything except quantum mechanics -- pretty crazy complaints!. After the dust settled somewhat, I was thanked by an Asst. Secretary of the Army for my work in promoting this work, which led to use of these simulations in training.
Clearly, the overwhelming majority of other reviewers of my work, who endorsed publication, have at the least established that these fewer critics were uninformed. Such meritless and meretricious reviews only bring into question the integrity of the reviewers. As I am quick to point out, these people were in the minority of reviewers. Nevertheless they represent a set of people that should be excluded from any serious reviewing of just about anything, but especially any interdisciplinary research.
In a given focused discipline, like nuclear physics, most often authors get pretty fair reviews, albeit this process can get strained a bit when a reviewer's own grants and contracts are threatened. In martial arts you can at least try to defend yourself face-to-face against your opponent -- until you get to a high enough rank that you have to deal with their politics!. Interdisciplinary research requires much more of reviewers than just extracting their professional opinions on their own documented expertise in a given specialty. They have to be honest with themselves, and often with their colleagues, on just what they know and what they do not know. Good common-sense judgment (intuitive and probabilistic, tempered by analysis and experience), separate from previous expert knowledge often is required. Interdisciplinary reviewers must be prepared to acknowledge that they are not capable of reviewing a particular paper or project. My own experience with the frequent failure of interdisciplinary reviewing has led me to take care to honor this practice, and I often review papers and contracts for several institutions.
The only way to preserve integrity is to always tell the truth and to be true to your own intuitions and analyses, albeit this self-discipline most certainly will not make everyone happy.
Lester Ingber <ingber@ingber.com> Copyright © 1994-2008 Lester Ingber. All Rights Reserved.
$Id: ingber_projects.html,v 10.244 2008/05/07 13:35:12 ingber Exp ingber $