Re: storage of sensor data with Fourier transforms

Hello!

I have a potential situation where I will have a lot of sensor data coming
in very often. (every second or so) The sensor data is from physics type
measurements, and will normally follow a slowly changing pattern with
sinusoidal disturbances. The overall shape of the data is more important
than catching high frequency disturbances.

I had the idea of taking the Fourier transform of the waveform and storing
the waveform internally that way to reduce storage requirements. When I get
some free cycles, I may try doing this. I would want it to be done in the
database in such a way that the user can still join to a table using this
internal storage scheme.

Why am I mailing this list? I'd like to ask if anyone has heard of someone
doing anything like this. I did a small search of the lists and internet but
didn't come up with anything. I just don't want to re-invent the wheel.

Thanks for your time,
Nathan

--
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Got Mole problems? Call Avogadro at 6.02 x 10^23.

On May 4, 2007, at 10:13 PM, Nathan Buchanan wrote:

> Hello!
>
> I have a potential situation where I will have a lot of sensor data
> coming in very often. (every second or so) The sensor data is from
> physics type measurements, and will normally follow a slowly
> changing pattern with sinusoidal disturbances. The overall shape of
> the data is more important than catching high frequency disturbances.
>
> I had the idea of taking the Fourier transform of the waveform and
> storing the waveform internally that way to reduce storage
> requirements. When I get some free cycles, I may try doing this. I
> would want it to be done in the database in such a way that the
> user can still join to a table using this internal storage scheme.
>
> Why am I mailing this list? I'd like to ask if anyone has heard of
> someone doing anything like this. I did a small search of the lists
> and internet but didn't come up with anything. I just don't want to
> re-invent the wheel.

It's an interesting idea in theory (sounds like a simple low-pass
filter would do most of what you want) but it doesn't sound like that
great an idea in practice.

Sampling at 1Hz isn't going to generate very much data at all, unless
you're capturing ridiculous amounts of data at each sample point (I
have boxes averaging hundreds of pieces of data a second totaling 10s
of k, along with a bunch of index updates without breaking much of an
I/O sweat).

Unless you have some very tight storage constraints (embedded
controllers, flash storage, ...) you're not going to see a problem
with storing the actual data, and you're likely to regret both
discarding the original data and expending much effort developing
software low-pass pre-processing for the input signal.

(Processing it for analysis is a whole other thing, but even that
will be much easier to do if you can apply variants on your algorithm
to the untouched original data).

Cheers,
Steve

"Nathan Buchanan" <nbinont(at)gmail(dot)com> writes:
> I had the idea of taking the Fourier transform of the waveform and storing
> the waveform internally that way to reduce storage requirements.

Aside from what Steve said: The Fourier transform in itself doesn't
reduce data size --- it's N points in, N points out. If you want to
reduce storage requirements you have to resort to lossy compression, ie,
deliberately throwing away some data. The transformed data might be
more suitable for doing that (eg you can selectively discard
high-frequency components) but do you really want to? Usually the point
of storing measurements is so you can do unspecified analysis on them
later. Applying lossy compression will restrict what you can
(meaningfully) do later on.

regards, tom lane

On 5/5/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> "Nathan Buchanan" <nbinont(at)gmail(dot)com> writes:
> > I had the idea of taking the Fourier transform of the waveform and
> storing
> > the waveform internally that way to reduce storage requirements.
>
> Aside from what Steve said: The Fourier transform in itself doesn't
> reduce data size --- it's N points in, N points out. If you want to
> reduce storage requirements you have to resort to lossy compression, ie,
> deliberately throwing away some data. The transformed data might be
> more suitable for doing that (eg you can selectively discard
> high-frequency components) but do you really want to? Usually the point
> of storing measurements is so you can do unspecified analysis on them
> later. Applying lossy compression will restrict what you can
> (meaningfully) do later on.
>
> regards, tom lane
>

Thanks for the replies. It seems I need to examine my plan more closely.

Nathan