Give ‘Em What They Want

As a result of being an alert but obscure data scientist, I have seen many accounts of my illustrious fellows in the field writing and marketing frequently against Microsoft Excel and its use as a database among many executives. To be sure, there are many cutting edge efforts at data warehousing and visualization that spend a great deal of ink and pixel space mocking Excel as not even being a real database, and showing gorgeous visuals and graphs of many varied types, some with excellent scholarly pedigrees [1] and worthy arguments. Yet as a data scientist, I spend my workdays pulling data into Excel and working with it and making visualizations on it for managers and executives, as well as external parties who have custom requirements in the data they receive, despite the fact that I have a great deal of skill in creating visualizations in a broad variety of ways that I could use to gain insights and share those insights with others. Despite my own ability to use contemporary data visualizations, despite my own interest (if not particular skill) in working with SQL database languages, and despite the power of such tools, what is requested of me, without exception, is data shown in a spreadsheet for others to use.

What is to account for this disconnect? On the one hand, there are many limitations with Excel and it is quite true that it is not a database program at all. It is a spreadsheet program, albeit a very successful one, with all the limitations that entails in terms of the presentation and linking of data. Yet many of the new and shiny tools in data management and visualization, even as they actively campaign against Excel, as if they were engaged in a political campaign and felt the need to ‘go negative,’ fail to capture the sustained interest of executives who, in the final analysis, simply want the Excel spreadsheets they receive and manipulate to work better, and are not interested in massive change to their own habits, even as they loudly proclaim how flexible their organizations are and enforce a great deal of continual change on the people at the bottom levels of their organizations. Change and flexibility, like slavery to antebellum slaveowners [2], is often something that is good for other people. When it comes to how executives and managers want to deal with data, there is a great deal of resistance to change, and in this light campaigning actively against Excel may be counterproductive.

Yet it is an indisputable reality that changes in data visualization and the work of people like myself in companies all over the world will not be based on what is the best principles, if such can be determined, nor will it be based on the convenience or choices of individual contributors like myself, but it will be based on the choices of executives themselves, possibly with some managerial input, but not necessarily, and very unlikely with any input from ordinary frontline workers at all. In that light, it is easy to understand why companies seek to market their products in ways that strongly resemble the most unpleasant aspects of political campaigns, with marketing advertised as fact, advocacy presented as white papers or research, and negative campaigning made against others rather than a focus on positive advertising. Marketing the new data techniques is done in a political fashion because it is a political act, and requires that companies vote with their pocketbooks and their time spent in training people on the new cloud-based websites or the programs to use them, and continue to vote for them through ongoing development work as well as monthly subscription fees. The stakes are high, and Excel is the enemy for these companies, and yet these companies are often unable to vanquish the forces of inertia.

Perhaps it is wise to take a step back and examine what it is that is so appealing about Excel in the first place, despite its manifold limitations. What is it that managers and executives like about Excel that other visualizations fail to provide. For one, people like the ease of pulling data out with filtering options that does not require using technical languages like SQL, and the way that once the data is pulled out, it can be worked with in almost a tactile sense. The managers and executives I am familiar with do not want to see a static design, and certainly do not trust the data, or the person working with it or presenting it to them, simply to have done it right. Even if their own skill at working with data is modest, or even close to nonexistent, they have many questions, and very good questions, about the data that they are given and why it does not often match, and why there is not one version of the truth that is consistent. They are irritated by the way that data can be input incorrectly and how it can be corrupt and how it can be corrupted. It is entirely proper that they should be concerned about these matters, as they spend their lives making serious and momentous decisions based upon the data that they receive. They have every right to want that data to be accurate, and to be accessible given their technical limitations and lack of time/interest in acquiring high levels of technical skills when that is what they hire data scientists for in the first place in order to outsource this worthwhile but irritating task.

What implications does this present for contemporary data firms which are looking for legions of enduring paying customers? For one, it suggests that many of them are going about matters all wrong. Their development of powerful software which can do amazing things but requires detailed and substantial expertise in esoteric computer languages is often misguided, because it cannot be used or even well-understood by those who make the decision to buy or to cut off subscriptions. Their mocking of Excel is beside the point, because what many of their customers want is not a solution that will require weeks or months of professional self-education in order to understand what a given software offering does, but they want something that will do what Excel does, be as simple to use and accessible to people who want to filter and sort and maybe make a pivot table, and to grapple with their data for themselves, rather than see it through a window in a black box that is comprehensible only to their technically-inclined subordinates several levels down the corporate food chain whose expertise is valued but not fully understood and thereby mistrusted. What is needed is to marry the powerful analytical techniques and visualization prowess of the best data services with interfaces that are simple to use and that provide access to the raw data so that it can be seen for itself in all its splendor, and preferably in a handy spreadsheet. Perhaps such a solution is boring, but that is ultimately what the people want—at least the people making the decisions and paying the bills. Why not spend more effort giving people what they want, rather than trying to convince them in vain that they want what you have to offer, and that they really do not need the simplicity of and ability to manipulate raw data that they so deeply want?

[1] See, for example:

[2] See, for example:

About nathanalbright

I'm a person with diverse interests who loves to read. If you want to know something about me, just ask.
This entry was posted in American History, History, Musings and tagged , , , . Bookmark the permalink.

6 Responses to Give ‘Em What They Want

  1. Pingback: Some Minutes Of The SQL Brown Bag Lunch #3 | Edge Induced Cohesion

  2. Pingback: Book Review: Your Code As A Crime Scene | Edge Induced Cohesion

  3. Pingback: Book Review: Draw To Win | Edge Induced Cohesion

  4. Pingback: Book Review: Driving Technical Change | Edge Induced Cohesion

  5. Pingback: Don’t Conduct The Autopsy Until The Technology Is Dead | Edge Induced Cohesion

  6. Pingback: The Most Attractive Celebrity You’ve Never Heard Of, Or Don’t Play That FOMO Game With Me | Edge Induced Cohesion

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s