Since the late thirties, after von Neumman formulated what is now considered the standard mathematical framework for quantum mechanics, many people have asked the question (Why complex Hilbert Spaces?).An obvious answer is:
- Both Schrodinger and Heisenberg used prototypes of separable complex Hilbert Spaces to formulate their respective approaches.More precisely,Schrodinger`s wave mechanics employed the space $L^2( R^2 )$ of square integrable complex functions,whereas Heisenberg`s matrix mechanics employed the space $ l^2$ of square - summable sequences of complex numbers.At the age of just 23,von Neumman realized that the spaces $L^2( R^2 )$ and $l^2 $ had something deep in common!That deep thing was the mathematical structure of a separable complex Hilbert space,and so the unification of Schrodinger and Heisenberg`s formalisms into a single mathematical framework was born by employing an abstract separable complex Hilbert space.Since then quantum mechanics enjoyed a special status in modern theoretical physics,as thousands of its predictions have been experimentally confirmed.
Note : A famous alternative to the Hilbert space approach is the algebraic approach, in this approach one starts with an abstract unital C* - algebra $A*$.Roughly speaking if $A*$ is commutative then we`ll have classical mechanics , whereas if $A*$ is noncommutative we`ll have quantum mechanics! There are other interesting approaches as well ( e.g,convex set approach).
Now given the experimental success of the Hilbert space approach in the last 85 years,one may ask, why should we change such a powerfull formalism that has given us so many correct answers ? Even more worrying, one can easily take the "shut up and calculate "attitude that many physicists take in regard to the foundations of quantum mechanics! However,if one is driven by foundations of physics ( for example Hilbert`s sixth problem on axiomatization of physics),then one may consider doing the following :
-Construct a brand new abstract axiomatic framework from which one can derive the Hilbert space formalism (and others including classical formalism) as special cases of this abstract framework.
I am currently writing a preliminary draft called " Quantum Formalism From Three Abstract Axioms ", hopefully I will post it here in near future! Do you think there is the need to change the standard Hilbert space formalism ? Feel free to comment!