Fast Fourier Transform - Cycle Extraction - page 4
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Practical solution
Thanks Simba for the info... no disappointment. At the very least I will learn something, and hopefully everyone who is interested in cycle analysis will. I am conviced to the core of my soul that cycle analysis can be used as a major part of any trading system. But as with anything, there is a right way and a wrong way to do it.
The CycleTrends v4 program is not the only program on the block. I've seen another half-dozen that look just as impressive if not more than CycleTrends.
There is a lot to learn, but I think it will be well worth it.
You and I are not that far off track really.
What I have done is write an indicator in MT4 that will remove the noise with an offset moving average as per Brian Millard, and export that to my external windows program that I wrote to continue with the analysis.
I will call FFT from my external program probably several hundred times in order to extract the cycles from the data. The problem that I saw was, the FFT gives different cycles for different data lengths of time. For example, a set of 5 highest amplitude cycles are different for 1000 bars and 2000 bars. So, what I want to do is step my data history starting at 500 up to 5000 bars, by 100, performing an FFT each time, then analyzing each set, throwing out the excess cycles that are close to being equal. That set is the set I wanted to perform a Bartels test on to verify the fit of 5 or 6 highest amplitude cycles. No, its not going to be perfect, but in theory you should at least be able to forecast a direction, and an approximate time for that direction to end.
All the above, I can do with my external-to-MT4 program. But I cannot find out the math behind the Bartels test, that I can put into a program. And, that is the key really. I'm not a math major, so I need to be able to use math functions that are built into most programming languages.
Yes I have seen the Goertzel indicator. I can easily add that to my external program for automatic analysis of any timeframe. I am contemplating that...
CarlHi,
For people following this thread and not yet experienced with cycle terminology...High Bartels test`s results for a specific cycle means that this cycle has a high probability of being "non random" ,so,it is fairly probable that we will find it in Future data too,and consequently we can use it to forecast evolution of future prices with a good probability of success(in theory).
Carl,I will try to look for the Bartels test formula and attach here ,in the meantime I have 2 suggestions for you.
1-Check for those cycles that are repetitive among a walk forward run of x/5000 of your dataset(If I understood you well ,this is more or less what you are trying to do with x=500 and a walk forward step of 100)...the common ones to most/all the steps..will,for sure have a high Bartels result.
2-I am working with Goertzel at the moment since I find it extracts the cyclic component of forex data fairly better than Mesa or FFT,and ,for example,while doing an analysis of GBPJPY...720 sets..for D1, H4,H1,M30,M15,M5,M1...I have found that all the datasets(different timeframes) had,among others,the following cycle periods...11,18,29,39...with good amplitudes at ALL timeframes...so,I don`t need a Bartels test to use JUST those periods as the basis of developing my cyclic indicators since I have found them to be permanent among a very wide span of time at different timeframes....My suggestion is that you could do a similar exercise with FFT and your favourite Forex pair,and see if you find anything similarly interesting.
Yes,there is a lot to learn..and,yes,it is worth every hour we invest on it,if I find the Bartels formula, I will post it in this thread.
Regards
Simba
Simba,
It's been a long time! For a while i didn't see you on the boards. Glad you are doing well. In response to your remarks, when you say "720 sets" among those time frames, what exactly do you mean? I was assuming it was something along the lines of using different amounts of bars per time frame or different time periods with the same amount of bars among the different time frames. I hate to assume though...:-) You know what they say after all....
All the best,
cl
Assuming
Simba,
It's been a long time! For a while i didn't see you on the boards. Glad you are doing well. In response to your remarks, when you say "720 sets" among those time frames, what exactly do you mean? I was assuming it was something along the lines of using different amounts of bars per time frame or different time periods with the same amount of bars among the different time frames. I hate to assume though...:-) You know what they say after all....
All the best,
clClahn,
What has 16 legs,14 testicles and 2 tiny breasts?
Yes assuming is probably going to distance you from the right answer..so,just asking..What?..will get you the right one fast and easy..Snow White and the 7 dwarfs
Use the Goetzel indicator I posted with 720 periods,not 500.
Regards
Simba
haha....glad you still have your wit....glad you are still using cycles....i've been using DF's for a very long time now with a lot of success....i just find i can "feel" the markets better....
best,
cl
This is an interesting thread! I have considered using FFT for cycle analysis, however the repainting and constant monitoring of the repainting are 2 things I don't like. I did like michal's point, and I wonderd if detrended FFT analysis would be better, but MESA seems more definite.
SIMBA, thank you for the link and indicator, can you say whether the Goetzel indicator repaints? You also mention it is better than MESA, how do you calculate it? We have the digital filter program from newdigital, is there another program you would use?
If you happen to find it I would like to see how bartels are calculated, but you have very easy to understand explainations
Thank you to everyone contributing to this thread
Facts
Codobro,
The Trouble with Facts is that there are so many of them...;)
I don`t know exactly how Bartels is calculated I just know it is a test for statistical significance,my own experience using just high Bartels cycles has been very negative,probably because I used them with a software cycletrends that used FFT,so there was continuous repainting of even the high bartels cycles.
Mesa has a problem,it is very bad on noisy data,and has an advantage,it is very good on low noise data,while Goertzel is very good on noisy data(see the papers at meyersanalytics for a detailed explanation)...so,since Forex data is noisy,I prefer to use Goertzel and then see confirmation by Mesa just to have a fuller view.
If you check this link SSA-MTM Toolkit you will see the right way to use mesa...what they do is first use SSA(singular spectrum analysis) to eliminate noise,then reconstruct the time series,without noise,then apply Mesa to it...there is exceptional free software there,basically by the Geophysics dpt of UCLA in cooperation with European CERN and a very detailed explanation about how to use the different cycle extraction models in combination to achieve the best results.
I have been unable to install it on my computers(windows xp and windows vista),so,if anybody tries it and knows how to install it,please be so kind to post it here.
OK,back to Goertzel...
The Goertzel indicator doesn`t repaint,you have to learn how to read it,what it gives you are numbers,forget about the time location of the curves you see...
Just point your cursor to the highest indicator top,normal non squared mode,it will give you back 3 numbers: phase,amplitude and period...when you see that period is,for example 0.0250..what do you do?
P.S:You will have to open the data window for this.
Well,actually, what this is telling you is that the highest amplitude cycle has a period equivalent to 0.0250 of the dataset length...since you used 720 periods..that means a cycle length of 18 periods(720*0.0250=18).
So,if we do it for EURGBP H4 now with squared=false and periods=720 we will get a highest peak with the following data:
amplitude=5.2547
phase=2.0174
period=0.0507
That means that the dominant cycle has a periodicity= 720*0.0507 approx 36/37 periods and that this cycle has a phase 0f 2.174..now go back to your digital filters generator and do the same analysis for 720 datapoints...
What you will see is the following:
1-A confirmed peak at 35/36 periods,which is similar enough to our Goertzel 36/37 periods...so,you can be fairly sure that there is a cycle there with that periodicity(36/37) that is not the byproduct of random noise.
2-Several higher peaks at periods from 5 to 28...What Mesa says are important cycles...Goertzel says are just the result of random noise,with the exception of the MESA peak at 23 that seems confirmed by a good amplitude Goertzel of 0.0356*720=approx 25 periods,so,you can use a 25 periodicity too in addition to the 36/37.
3-A very high peak at 106 periods...well,you are using 720 datapoint,so a 106 cycle period looks like probably random...you can then use Mesa with full history and see if it is still there...No,it isn`t,so,you discard it.
Now you know that to use cycles with EURGBP H4 you have to use 36/37,25 and 49/50(you will discover it if you continue the analysis) periods.
Now..a very good question to ponder...Does the phase being at 2.0174 has some relevance? Don`t be shy and contribute your opinions
Regards
Simba
Download link:
SSA-MTM Toolkit Downloads
0.Install Ubuntu 64 bit
1.Download Spectra_st_linux64.exe
2.Install Grace
3.Run Spectra_st_linux64.exe
3.Unpack soi.zip
Variations of the Southern Oscillation Index (SOI)
between January 1942 and June 1999. Time on the abscissa is
in calendar years, and SOI on the ordinate is normalized by its
standard deviation.
4.File/Data, Read vector, soi, Read File
5.Plot,Vector
http://www.atmos.ucla.edu/tcd/PREPRINTS/2000RG.pdf
Thank you SIMBA,
My first question is why have you chosen 720? You had mentioned using 200-210 bar samples previously. What do you think of using around 11.08.08, the last trough?
I have loaded the Goertzel indicator, squared=false with a 720 sampleset. Measuring from the highest amplitude line:
2008.06.08 16:00
Amplitude 5.2746
Phase 2.1042
PeakPer 0.0514
The amplitude fits within yours, and the peak from the sampleset looks within yours as well. What of my phase reading? It looks dramatically different, though I could understand having a larger room for discrepant phase value. Would you consider this too much discrepancy for effective application?
I certainly think phase has relevance, allowing for attenuation. I'm thinking about Ehlers mention of recuding phase lag. How would this be properly applied?
I have 36.504, which corresponds to the spectral analysis of 720 inside the digital filter program.
However loading all data shows that as a trough, as well as 25. 47 seems to correlate in the same way, but with near 0 amplitude. Very interesting numbers, I used 660 bars as well, 25 and 47 seem to show up, and 35/37.
Ssa-mtm
About the toolkit, barnix posted a good use. Here is a little explaination:
SST-MTM is a toolkit, like an addon, for a program like Grace.
Grace has to be compiled to run, which is for Linux, but can be run under Win32. Grace FAQ (for Grace-5.1.22)
Cygwin is a program that runs Linux functions under Win32 and grace comes with it. Cygwin FAQ
If you do get this, a few things to note about installation:
Do not install to C:\cygwin for example. Instead, use something like C:\CYGWIN_INSTALL\Cygwin and use this for local install as well. It will prevent conflicts from Window's root files.
Under Install options, Catagory, the X11 Catagory has Grace as an optional install.
You'll need Spectra_st_linux32.exe if you plan to use this method. It seems less complex than installing another OS.
Thanks for fuel
Variety of sources is the fuel of achievement...;)
Codobro and barnix,
Thanks for your comments on the SSA toolkit installation.
Codobro:Just try it and see if you find it useful,different datasets might give us slightly different phases...if you have similar period and amplitude,that looks robust enough to be useful..
My question about phase was regarding to its uses as a trigger...where is the trigger point?
720 is a good number when working with cycles,since essentially we are trying to convert price changes into degrees of a circle that moves forward in time,so 360,720,etc are good starting points...I have tried to use 1440,for h4 and couldn`t,so I stick to 720....You can use 500,450 or whatever you want,there really isn`t any rational reason to stick with 720,besides the fact that,in my experience ,I get better results with that period...you just try and decide about your choice period....since Goertzel works well with noisy data the longer the dataset the better....if the indicator worked with 7200 points,I would run a walk forward in steps of either 720 or 360 periods and check for persistence between the full series(7200) and all its 360/720 subsets...the problem is that the longer cycles probably will not appear on the 360 subsets
I have used and still use 200,250 bars with Mesa to see if they confirm or not the full history analysis..basically do this kind of analysis for 5 September onwards (GBPJPY,GBPUSD,etc)to see if there are hints of a change in paradigm as price evolution may suggest,but this is a secondary type analysis,when it strongly differs from full history analysis you can run 2 simultaneous cycle models and either stay out of the market or decide which of the 2 cyclic models(full history or 200/250 bars) is more in sync with the market....of course filtering this short term analysis with Goertzel helps in eliminating the random cycles from the equation.