****ing 2 Gig file size limit

[KrazyHorse]

Originally posted by Jon Miller
I hate that it is interpreted (Always compile it, always).

Indeed.

And that it is mostly done by physicists and so has all sorts of not very well documented features/etc.

That's putting it mildly.

However, I have been using it since I became an experimentalist and so know it quite well.

I like that it has a lot of handy classes/etc for histograms/plotting/fitting/etc.

I know, I know.

[KrazyHorse]

Originally posted by BlackCat


Well, then you really are interested in some kind of data compression - not to save space, but to improve performance of the analysis. That can't be done by zip, gz or the like - only through good programming.

You honestly don't know what you're talking about. The type of data he's talking about cannot be meaningfully compressed.

[BlackCat]

Originally posted by Asher
It's never good to have the "database handlers" decide logic on how to set up your own tables.

DBAs should only do routine maintenance ****, they should not be making design decisions. The developers who know what they need done should be making those decisions.

We had our DBAs restart the things if they went down or perform regular maintenance. They could NOT touch our tables/data/schemas without explicitly asking us.

Asher, are you drunk or what ? I said I made the design decisions, then the DBA must do the optimizations that oracle offers.

Though, I usually tells wich indexes that at least should be made (there rarely is need for other).

[Asher]

Originally posted by BlackCat


Asher, are you drunk or what ? I said I made the design decisions, then the DBA must do the optimizations that oracle offers.

Compression is a design decision. That's why I'm saying you're making no sense.

Stuff like compression and indicies (clustered vs non-clustered, for instance) are critical design decisions.

[BlackCat]

Originally posted by KrazyHorse


You honestly don't know what you're talking about. The type of data he's talking about cannot be meaningfully compressed.

Well, if every mesaurement is unique and differs from it's neighbours and it's a continious stream, then it's difficult to compress. Though, it would surprise me if it's purely random.

[Jon Miller]

Originally posted by BlackCat


Well, if every mesaurement is unique and differs from it's neighbours and it's a continious stream, then it's difficult to compress. Though, it would surprise me if it's purely random.

Yes, Yes, Yes, Yes, but not purely random, which is why we spend years analyzing it to make a measurement.

JM

[KrazyHorse]

Well, if every mesaurement is unique and differs from it's neighbours and it's a continious stream, then it's difficult to compress. Though, it would surprise me if it's purely random.

It is a PHYSICAL PROCESS

It is the DEFINITION of random. There is probably significant correlation between the measurement of neighbouring cells, but that only gets you in the ballpark. The digits after the leading one are essentially independent. If they weren't then there would have been no purpose in building a detector with such fine resolution.

[BlackCat]

Originally posted by Asher

Compression is a design decision. That's why I'm saying you're making no sense.

Stuff like compression and indicies (clustered vs non-clustered, for instance) are critical design decisions.

I think that we have had a similiar discussion a couple of years ago - we store the raw data in oracle but have the seach engine working on a non oracle database. Oracle has improved but can still not compete.

[Kuciwalker]

If KH could compress his dataset he'd probably be better off just taking less data.

[Asher]

KrazyHorse, Joan Millar:

your thoughts? http://www.cnn.com/2008/POLITICS/12/...rgy/index.html

[Boris Godunov]

I like the picture of him explaining how to escape from a Chinese Finger Prison.

[Jon Miller]

He is definitely very smart, and his position as director and working with lots of different groups (Which I didn't know about until I started looking up information about him) suggests that he can handle politics.

JM

[KrazyHorse]

Originally posted by Kuciwalker
If KH could compress his dataset he'd probably be better off just taking less data.

Yes, that's the point (note this is not actual experimental data)

[BlackCat]

Originally posted by Jon Miller


Yes, Yes, Yes, Yes, but not purely random, which is why we spend years analyzing it to make a measurement.

JM

Well, advanced data compression isn't just "squeezing" the zeros - it's also turning streams of data into something shorter.

That said, I will allways recomend keeping the original data - besides errrors in sw, others may find unexpected things not sought out by the initial researchers.

[KrazyHorse]

Well, advanced data compression isn't just "squeezing" the zeros - it's also turning streams of data into something shorter.

BC, do you mind explaining to me how you would meaningfully compress 2 uniformly distributed 10 digit decimal numbers with correlation coefficient 0.5?