DPlot

[raswaim]

I tend to run a lot of huge data sets; plots with more than 20-30 million points are not unusual, which is one of the reasons I like DPlot in the first place, because it handles these sizes well. However, I do have one annoyance that I run into periodically when I'm really pushing the limits... It occurs when you have a large amount of data loaded and you try to do something like "Generate..." a new curve (beyond the number currently allocated). If I do happen to run out of available memory when the new curve is being allocated, I get the warning that generally says there's insufficient space for the <billions and billions> of points I'm trying to use, and to please consider erasing a curve or asking for some more reasonable action.... However, at that point the re-allocation attempt has already destroyed the content of the curves; they're still there in name and have all of the dimensional information, but the X and Y data values (should have mentioned they're 2D XY plots) are all zero. Is there anyway the behavior could be changed to not do this? I realize you're probably somewhat at the mercy of a system call and also need to recover the existing space to get the quantity of memory needed (which is why I classify this as a suggestion), but if there's any check that can be performed before data loss it would be helpful. I'm using Win7 64-bit if that makes a difference, but I think the behavior was the same with XP and earlier DPlot releases.... Thanks - R
_________________
- RAS

[DPlotAdmin]

Richard,
I'll have to look into this, but what you described should not happen. DPlot first attempts to allocate new arrays, and if there is insufficient memory it will save the current arrays to a file on disk. If saving the data to a file fails (insufficient disk space, for example), then you should get a message to that effect and the original data will still be OK. If saving to a file works, DPlot destroys the original arrays and attempts to allocate new arrays again. Whether it works or not, the original data should be restored from the file that was saved.

From what you said my guess is either the data was not written properly for some reason or not read properly, though I can't think why that would be. The temporary file is saved to whatever you have the TEMP environment variable set to. Typically on Windows 7 that will be C:\Users\<username>\AppData\Local\Temp

If anything in this description hops out at you of course let me know. Meanwhile I'll put DPlot through the wringer here.
_________________
Visualize Your Data
support@dplot.com

[raswaim]

Interesting. As far as I can recall I've experienced this behavior before on another machine as well, and I'm not sure I've ever seen the design behavior you describe (not that this happens all that frequently). The temp directory exists, and there's plenty of space. I see some files in TEMP that have DPlot signatures inside them but none that are close to the size of the data I was manipulating. As best I can recall I was attempting to use Generate->Integrate a curve when it happened. There were around 7 x 3 million point curves already loaded. The 7th would have been created by a Generate->Differentiate and then y-range clipped, so there would have been another successful re-allocation of memory to get me to 7 curves from 6 just two commands previous to the ultimate "insufficient memory event".

One thing that occurs to me that is probably different than your normal operation is that more often than not my data sets are loaded by a custom plugin for my file format. Is it possible there's an issue with the original creation of the structures that makes it work OK for everything else I do but fail the re-allocation protection (but not a successful reallocation)? I wrote the fileimport plugin more than 7 years ago and haven't touched it since; there are several other folks here who've used it over the years and not reported any issues, but its certainly a difference from your experience base. The other thing is that DPlot and my machine both are signed up for marathon sessions; I may run a few hundred plots through it over the course of days without closing DPlot or rebooting the machine. There's almost always a DPlot window with a couple plots submerged somewhere on my desktop....
_________________
- RAS