visualdatatools.com Forum Index visualdatatools.com
Discussion for DataTank and DataGraph
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

FFT and data filtering

 
Post new topic   Reply to topic    visualdatatools.com Forum Index -> Feature Request
View previous topic :: View next topic  
Author Message
okonkole



Joined: 01 Jun 2015
Posts: 10

PostPosted: Wed Mar 23, 2016 8:28 pm    Post subject: FFT and data filtering Reply with quote

Could you implement a new command to calculate the FFT and maybe also other type of filtering (low/high pass) for a given signal (x,y).
Maybe it could be similar to the fit command where you chose your fit to apply to the x-y data (but on this case you chose your filter).
Back to top
View user's profile Send private message
David
Site Admin


Joined: 25 Nov 2006
Posts: 1964
Location: Chapel Hill, NC

PostPosted: Sun Mar 27, 2016 12:26 am    Post subject: Reply with quote

It might fit in there (ha ha), but initially I would like to add it to the Plot Action column. This column asks for (x,y) columns and the computational method. You can then draw the result as a function of x.

Can you tell me (here or through e-mail) what action you want to apply?

Standard FFT requires the x values to be uniformly spaced, even though that can be relaxed. What convolution do you have in mind? And do you expect the data to be periodically replicated (for an exact filter)?

David
Back to top
View user's profile Send private message Send e-mail
samcub40i



Joined: 29 May 2017
Posts: 11

PostPosted: Mon May 29, 2017 10:37 am    Post subject: Reply with quote

This would be useful if it could be implemented as a periodogram with windowing options.

One issue I have though is I have had a need to process rather large data files (over a Gb) to develop spectra that may only be comprised of 512, 1024 bins etc. Rather than attempting to load the entire file into datagraph at once just to reduce, it would be nice to tag a large csv dataset with a taylored import method (perhaps display first N lines of data), then apply a process like the periodogram that works on the whole file.
Back to top
View user's profile Send private message
David
Site Admin


Joined: 25 Nov 2006
Posts: 1964
Location: Chapel Hill, NC

PostPosted: Wed May 31, 2017 10:08 am    Post subject: Reply with quote

Can you give me more information (here or through e-mail)? I'm not clear what you mean by a >1GB file going down to 512 bins.

David
Back to top
View user's profile Send private message Send e-mail
samcub40i



Joined: 29 May 2017
Posts: 11

PostPosted: Wed May 31, 2017 4:23 pm    Post subject: Reply with quote

For example, I might have a csv file with columns of time history signal data with over a million rows. A periodogram (or spectrogram) would perform a FFT on 2^N (i.e. 512 or 1024) rows at a time with overlap and window functions. The FFTs are then averaged so the final output is only 2^N rows in length.

There are other codes that perform an FFT on the whole signal at once, but this is not always what is desired. And since very large files may not be practical to load entirely in memory, being able to create an import template to handle header information and formatted date columns for example would be beneficial. Formulas, plot commands, etc. could be suppressed, or restricted to a smaller sample until all was ready to tediously read through the large file once.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    visualdatatools.com Forum Index -> Feature Request All times are GMT - 3 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group