The Dog Ate My Data

The line to remember from the discussion at finds2011 was “The dog ate my data.” One of the stated goals of Finili is to produce more accurate results. One of the classic errors in handling data is leaving out part of the data. We heard some wild examples of ways this can happen. The one that sticks in my mind is the story where one disk was missing, perhaps stolen, from a disk array, resulting in the loss of the records that were stored on that disk. There were many other stories, but from the end user’s point of view, they were all variations of the same excuse. You might as well be saying, “The dog ate my data.”

Regardless of the specific cause, errors won’t slip through undetected so often if there is a way for the programmer to immediately see that something is off. We think we can make that happen with real-time data visualizations that display while the program is processing the data. These would take the place of the hourglass icon that some programs display while they are running. They would show, in a compressed form, what the program is doing with the data, like an oscilloscope, or perhaps, as someone suggested, as a kind of live infographic. Infographics are part of our objective anyway, so why would we withhold them until the end of the process?

There were many signs of progress at finds2011, but the boldness of this one discussion sums it all up. If a group of people can discuss Finili and the fine points of the way it might work, it shows that the design of Finili is far enough along that it is starting to look solid. The most difficult work might still be ahead, but at least at this point we are all assured that we are working on, and talking about, the same thing.