In this part of the discussion, I’m going to present an example of how CBE thinking interacts with a real-world problem. As part of the recent LinkedIn dialogue, I tried to think about how CBE might approach one of my own experienced areas of competence – data analysis. Our personal case studies (N=1) are always good sources of data – it’s guaranteed to be rich, although its generalizability isn’t always equally applicable. But it’s a starting point.
I’ve been a practicing data analyst for going on 40 years now. It’s a learned skill. I was over 30 and back in graduate school at the University of Michigan before I discovered it. The first step was taking a few courses (part of my formal program) that taught me some basic vocabulary, some of the formal rules, and a few specific procedures. I did very well in these courses; if education had been the defining assessment mechanism, I would have then been declared competent. But when I went out of the educational crèche into the world of real research, I soon found out that I knew virtually nothing about analyzing real data (“You know nothing, Jon Snow!”)
Starting partway through my coursework, I informally apprenticed myself to one of my professors known to be a good data analyst with a lot of research to his credit. I accomplished this mostly by walking into his office every other day or two and asking if there was anything in the works that I might be able to help him out with. Eventually he allowed me to do some data coding, then run a few statistical tests, clean up the office, walk his dog, and talk with him about the study and the data and what he seemed to be finding, and why. And then even to suggest a few tweaks to the study and a couple of areas of exploration that he hadn’t considered. Eventually I even got third-author credit on a couple of articles. I was hooked!
So much for the first three or so years. I won’t bore you with the details of the next several stages; just note that they involved getting connected (again by informal means) with a world-class researcher named Everett Rogers, working with him on several of his projects and eventually being co-PI with him on one; as a consequence of that project, working for NSF where I had to assess the merits or demerits of proposed research and be able to talk with academics on an equal if nor superior level about every aspect of a project; entering academia teaching organization issues but being drawn into teaching research methods; working with RAND Corporation as a consultant on a lot of projects; proposing my own research to Federal agencies and carrying out a number of studies; going to another institution where I reorganized the research training component of the PhD program from the ground up, to initial skepticism and later credit; and eventually transferring again where with a couple of other folks we essentially created a very fine all-online PhD program from raw scratch.
The point here isn’t what a wonderful data analyst I am; it’s simply that I’ve been involved with data and data analysis for a long time. Clearly, when I started I was incompetent; I believe that I’m now entitled to describe myself as competent. The question is just when in this progress of years, roles, places, and data did I pass into competence?
But, one might ask, is “data analysis” really a competence as such? Might it not be made up of a whole lot of smaller domains that collectively practiced might add up to data analysis? Presumably we could break out sub-tasks like survey design, sample selection, data cleaning and management, choosing statistics, interpreting statistical output, preparing informative reports, etc., and develop separate assessments for each of them. Your overall data analysis competence would then be a function of how many of these smaller certifications you managed to corral. In theory, any complex activity could be broken down in this way.
But the problem here is complex activities are complex. The whole is more than the sum of its parts, and unless all those smaller tasks are coordinated and meshed just right, the entire enterprise is at risk. Not to mention that there are so many possible ways of categorizing the activities in data analysis that developing a list of activities for separately certifying competence in them would be both incomplete and superfluous. A skill that’s vital to one project may be useless to another. And each project is likely to weight the importance of different skills differently. You hire someone because they can analyze data, not because they can perform any number of smaller tasks. Serious organizational competencies are much more holistic than reductionist.
In general, trying to reduce a complex holistic skill to a set of easily measured activities is impractical. While there are a lot of specific component activities someone can perform during a data analysis, few of them mean anything apart from the whole context of the project. Probably well over 50% of them could be performed by a trained monkey; they are the simple but necessary housekeeping that underlies good analysis. It’s probably the last five or so that make someone a competent data analyst – things like figuring out analytical strategies, interpreting variables, assessing generalizability, and the like. Competence in these areas is very hard to measure because it’s so situational. Things that work for some kinds of data don’t work for others; there aren’t many routine procedures at this level of analysis. Moreover, most projects of any significance involve several kinds of data, with different rules applying. Choices here are much more craft work than production, and it’s a characteristic of craft output that it generally can’t be precisely and replicably measured. It’s the difference between scoring a math workbook and judging the quality of pies at the country fair. Both are grading tasks, but the latter is a lot more complicated than the former, and I’d hate to try to make up a precise assessment sheet or rubric for it.
There’s no question that there are a lot of skill areas that can be characterized and measured and competence levels assigned. (I’ll pass for now over issues of stability, reliability, and consistency or performance in context, but hold those thoughts.) But particularly as one moves beyond the more mundane parts of the world of work, jobs are increasingly complex and holistic, and assessment becomes more synthetic than analytic. Being able to do various things is important; but more important is being able to put things together.
I’ve talked here about data analysis because I see it as a sort of paradigm for meaningful work. One can be very good at it, or not so good. If you want to call being very good “being competent” or “demonstrating competency”, fine. But there really aren’t any measurements bisect the different arcs of skill development I briefly noted above that allow one to find the “incompetent/competent boundary”. Things matter in very different ways at different times, and context accounts for much of the interpretation.
If data analysis is a paradigm of an important organizational skill, one that needs to be learned, it points to some limits to the CBE model. While it’s pretty easy to tell good data analysis from bad data analysis, it’s much less easy to trace lines of competence. And if that’s true, then CBE, with its emphasis on competency certification, isn’t going to be able to help us unravel the issue of improving employability. And I’m afraid that we may be over-promising things that some internal contradictions in the models themselves won’t allow.