****ing 2 Gig file size limit

[Asher]

Originally posted by snoopy369
Oh, in my case certainly. Unfortunately while actually USING the data, that's not an option...

(By optimization, I'm talking about 8GB -> 14MB ... bleck ... )

With the power of today's CPUs, in memory compression is more than an option.

How do you think game consoles make use of their 512MB of RAM? They have a thread or two whose entire purpose is to compress/decompress memory as it is about to be used or put away.

[KrazyHorse]

Originally posted by Asher
I bet if you zipped it you'd see tremendous optimization in size.

If you're talking to me, I doubt it. Most of the data consists of almost random strings of digits.

[BlackCat]

Originally posted by Asher
I bet if you zipped it you'd see tremendous optimization in size.

Yup are quite right - zipping the oracle database will certainly improve performance

[Asher]

Originally posted by KrazyHorse


If you're talking to me, I doubt it. Most of the data consists of almost random strings of digits.

Was talking to the librarian.

[BlackCat]

Originally posted by KrazyHorse


Not one which would be helpful to you.

I might surprise you

[Jon Miller]

Originally posted by BlackCat


I assume that it was an answer to me (pretty quick )

yeah, it is relevant to know the zero's, but there are no reason to register them one by one - all you need to know is what areas/cubics (dammit, what's the right word ?) - that save space and improves performance. only problem is that it demands an intelligent programmer

I have many different channels per detector per event and many many billions of events. Thousands of detectors as well. I also have other bits of information.

JM

[Asher]

Originally posted by BlackCat


Yup are quite right - zipping the oracle database will certainly improve performance

You never mentioned it was in Oracle.

But I'm not sure if you're aware that Oracle does have table compression features? It has significant impact on database size and either negligible performance hits or slight performance benefits.

You don't seem to be considering the importance of compression which permits data to fit into memory/disk caches...

[Asher]

When I was at Lehman our trading database (Sybase) would be several terabytes in size if we did not use compression. It'd be unusable without it, really. But we had a beefy database cluster also..

[snoopy369]

Originally posted by Asher


With the power of today's CPUs, in memory compression is more than an option.

How do you think game consoles make use of their 512MB of RAM? They have a thread or two whose entire purpose is to compress/decompress memory as it is about to be used or put away.

Oh, the entire dataset isn't put into memory at any point, thank god. Only whatever current row(s) are being actually operated on.

I don't doubt that there are ways around that, but we don't deal with THAT insane of datasets often enough for it to be worth it. And, we don't use Linux, so we have NTFS and a virtually unlimited file size (for our purposes).

[Jon Miller]

BTW, we might be able to do greater then 2 GB now.

A couple links to the data storage we use:

A link to our CODA system (Data Acquisition): http://coda.jlab.org/event_format.htm

A link to ROOT, our standard analysis library/etc:

http://root.cern.ch/root/html/TTree.html#TTree:description

JM

[BlackCat]

Originally posted by Asher

You never mentioned it was in Oracle.

Yeah, I guess that you assumed it to be SQL Server

But I'm not sure if you're aware that Oracle does have table compression features? It has significant impact on database size and either negligible performance hits or slight performance benefits.

You don't seem to be considering the importance of compression which permits data to fit into memory/disk caches...

Uhm, no offense, but I have people that do optimazion on the database - I handle the logic. That said - it's always a good thing to know/talk with the database handlers.

[KrazyHorse]

Originally posted by Jon Miller
A link to ROOT, our standard analysis library/etc:

http://root.cern.ch/root/html/TTree.html#TTree:description

JM

God, I hate root.

It is quick, but it's slow and has weird behaviour.

Unfortunately it's what I'm using for this current thing.

[Asher]

It's never good to have the "database handlers" decide logic on how to set up your own tables.

DBAs should only do routine maintenance ****, they should not be making design decisions. The developers who know what they need done should be making those decisions.

We had our DBAs restart the things if they went down or perform regular maintenance. They could NOT touch our tables/data/schemas without explicitly asking us.

[Jon Miller]

I hate that it is interpreted (Always compile it, always). And that it is mostly done by physicists and so has all sorts of not very well documented features/etc.

However, I have been using it since I became an experimentalist and so know it quite well.

I like that it has a lot of handy classes/etc for histograms/plotting/fitting/etc.

JM

[BlackCat]

Originally posted by Jon Miller


I have many different channels per detector per event and many many billions of events. Thousands of detectors as well. I also have other bits of information.

JM

Well, then you really are interested in some kind of data compression - not to save space, but to improve performance of the analysis. That can't be done by zip, gz or the like - only through good programming.