Your Database Sucks

While I am aware that the sort of work I do in data analysis is not necessarily the sort of work that other people find to be the most exciting, it does occasionally lead to musing on my part. I suppose it would be most interesting for others if my work inspired poetry, but it would be the sort of poetry that would be extremely dull and esoteric, far less easy to comprehend than my normal poetry, of the kind that I am familiar writing [1]. That said, occasionally my work provides opportunities for me to discuss larger matters of societal importance that exist beyond myself and my narrow work, to the extent that they may suggest areas of larger interest. After all, I generally like to speak about matters that are of potential interest to someone else, at least, otherwise there would be no purpose for me to share my own thoughts aside from the therapeutic benefits of writing.

Several years ago, I read a book by the late and great film critic Roger Ebert which was called, not very charitably, “Your Movie Sucks.” This particular book contained a large amount of movie reviews for films that were not very good. Some of those films were not particularly memorable, and at one and a half stars out of four were rated slightly worse than average. These films did not receive a savage review, just a mixed to adverse one. It was the films that ranked at one star and half a star, though, that received the most blistering sort of reviews, some of them highly entertaining for me to read as a fellow prolific reviewer, and probably rather mortifying and embarrassing for the filmmakers, if they were (as so many creative people are) sensitive to criticism. This particular book had the upside of inspiring me to collect those negative book reviews that I have written under the title “Your Book Sucks,” if I can ever get around to collecting them into an e-book, per the wise suggestion of a former professor of mine.

At my job, I have to work with a particular database that handles our call data. Since the place where I work both calls out and receives a lot of calls, it is not hard to imagine that I have to sort through the data for tens of thousands of phone calls a month. I am aware that this sort of load can be somewhat taxing on even the best of databases, but this is not the case with the database I work with, which often takes hours to pull a single report about the calls for a single phone line, calls that it would be shorter to listen to with a stopwatch to count the time than it takes for the database system to work its way through the calls. Clearly, the database is more corrupt than a New Orleans police officer. These are not the only irritations one has to deal with, as there are quirks where the program pulls its date and time data in one window rather than having one window for the date the call was on, and another for the time of day that the call was on, which would be most convenient. Apparently this particular system does not like working in ways that are convenient and user friendly.

Of course, there are times, too many times, where the database does not work at all. Since my job depends on being able to pull call data reliably, the fact that this is not reliable can make a day a little more stressful than usual. Today, as might be imagined, was one such day. So far, four times today the database has gone down for significant periods of time, which is unacceptable. Several times the database went down while a report was being run, leading to delays and frustrations and wasted time. At least once it ended up in a call to the company that runs this particular shambolic database, with error messages like the following: “A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.” There are certain base expectations that one has of companies that are receiving a lot of money to run a database in an essential area of business, and one of those expectations is to work. If one was looking at six sigma level of reliability, this level of service would probably struggle to reach past one or two sigma, when five or six sigma would be the acceptable and expected levels of reliable service.

Nor is this pathetic database performance limited to work. Even games can be affected by such difficulties, which make them a lot less enjoyable to play. I once played an online game that was based on daily turns called Renaissance Kingdoms, where I played an English peasant from Chester who had a chequered background and was active in the church and politics named Metanoia (the Greek word for repentance [2]). The game was similar to others, in which every day in real life was a day in the game world, and where one’s tasks included working, traveling, and eating, with various items like weapons and armor that could give one better performance in a fight and certain statistics that could be raised through eating specific foods that could be earned through picking fruit or fishing, for example. The game, though, had a very poor database that was down for an hour or more per day, as it was hard for the people who made the database to keep it under control and to make it work properly, which made the game less enjoyable to play when things inevitably went wrong (when items disappeared from one’s inventory, for example).

Why is it so hard to build a stable database? It would appear as if data integrity is itself a major issue. One would think that one could design something that was more or less stable in its essentials and then work on supporting elaborations in terms of area or items besides that, but this seems far easier to do in theory than in practice. Ideally, the information that would be put in the database would be automatically generated and would not require a great deal of manual oversight. From personal experience, I know all too well that information which requires physical surveillance and editing tends to make matters more complicated and less stable, and also more time consuming to fix, while those things that are automatic via well-designed forms and queries is far less time-consuming and much less frustrating. We do tend to make life difficult on ourselves or others, though, and that is something that must be admitted by anyone who deals with data, or people.

[1] See, for example:

https://edgeinducedcohesion.wordpress.com/2014/08/19/are-you-gonna-cross-that-line/

https://edgeinducedcohesion.wordpress.com/2014/07/16/33/

https://edgeinducedcohesion.wordpress.com/2014/07/14/99-shades-of-gray/

https://edgeinducedcohesion.wordpress.com/2014/04/01/broken/

[2] https://edgeinducedcohesion.wordpress.com/2011/07/26/metanoia/

About nathanalbright

I'm a person with diverse interests who loves to read. If you want to know something about me, just ask.
This entry was posted in Musings and tagged , , , , . Bookmark the permalink.

5 Responses to Your Database Sucks

  1. It sounds like the lack of integration within your software programs is the root of your company’s IT evils. You need a wizard who can create a system to streamline their internal communication.

  2. Pingback: The Glory Of An Analyst Is To Analyze | Edge Induced Cohesion

  3. Pingback: Book Review: Company | Edge Induced Cohesion

  4. Pingback: Hurricane Maria’s Death Toll: A Statistics Story | Edge Induced Cohesion

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s