Model updating with terabytes of data in real-time
Readers of my Facebook page may remember that I’ve been writing about model updating high frequency trading data on an Intilop at sub 100-ns speeds. One of the tricks to doing this …… is that *all models must be updated in a single pass and the data must be thrown away*. Throwing data away sounds like heresy, but it is not.
On my Facebook page, I described how to take the mean and standard deviation of the data in a single pass, using a continuous update scheme. Mathbabe describes a similar scheme for model updating today. However, this scheme is complicated by the fact that lower frequency data has a different variance than higher frequency data and sometimes this matters.
I’m posting this note quickly this morning because I want to use it as a placeholder to discuss this in more detail in the future.
