10/6/2023 ☼ not-knowing
This essay is different from the ones that came before it in this series on aspects of not-knowing. The main difference is that I’m very actively working out the ideas below. An earlier version appeared in my newsletter on 17.5.2023; the version below has been revised and expanded with contributions from friends and discussants in my monthly public discussion series on not-knowing.
If you’re new to not-knowing (a concept different from both formal risk and from uncertainty), it might help to take a look at my introduction to not-knowing, my overview of overloading and appropriation and the updated diagram below before reading on.
With all that out of the way, my argument begins from the position that the word “uncertainty” is appropriated when it is used in a way that does violence to its meaning as unquantifiable not-knowing about a situation (i.e., Knight’s definition). For Knight, true uncertainty is not-knowing that is “not susceptible to measurement and hence to elimination.” True uncertainty is defined by the unquantifiability and unmanageability of not-knowing (as opposed to risk, which is not-knowing which can be quantified and thus managed and eliminated).
The appropriative strategy seems to be to use “uncertainty” to describe a form of not-knowing that can be quantified and thus can be made controllable. Examples from machine learning might illustrate what I mean by this.
One ML task is to develop a machine model that can evaluate a set of data (e.g. a bunch of photos) and put labels on them (e.g. “this photo is of a mug,” “that photo is of a twig”) — an image classification model. The fewer the errors in the output (e.g. appending the label “this photo is of a mug” to a photo of a twig), the better the model. Because the model is trained on some data (e.g. a dataset of images of mugs and twigs that has been labeled by a panel of humans) and then used on other, different data (images of things that could be mugs and twigs that need labeling), there can never be 100% certainty in a model’s labeling outputs.
When it comes to using a model, it’s important to know how much to trust the model’s outputs. This is particularly true if the outputs will be used to take actions that have important implications in the real world — such as an image classifier that is used by self-driving cars to detect and avoid traffic hazards. This model will have been trained on a dataset (say, of images of traffic hazards labeled by humans) and then used to label images taken from a self-driving car’s video camera. There won’t be 100% certainty in whether the model will correctly label a traffic hazard as “traffic hazard.”
In the ML context, this lack of 100% certainty would be labeled “uncertainty,” but what it actually seems to be is [not-knowing whether the model’s output is objectively correct or not, given a restricted range of assessments of correctness and strong assumptions about the similarities between training data and actual data being processed by the model].
This type of not-knowing doesn’t sound like either the narrow definition of formal risk or the overloaded informal definitions of risk, which may be why the AI/ML industry has decided to call it “uncertainty.” And this type of not-knowing about a model can be estimated and quantified (“uncertainty quantification”), thus permitting the model to be operationalised. One currently fashionable family of approaches for “quantifying uncertainty” in ML is conformal prediction, which “uses past experience to determine precise levels of confidence in new predictions.”
In fact, there are at least four different kinds of partial knowledge situations ML runs into:
[This non-exhaustive list was developed with the discussants in the fifth discussion on not-knowing.]
Of the four types of uncertainty in ML identified above, only the first two (uncertainty from poor data quality and model opacity) seem to have epistemologically viable technical solutions at the moment — and neither constitutes true uncertainty in the Knightian sense. I’m not sure that the latter two types of uncertainty in ML (uncertainty from inherent ambiguity in data and from subjectivity of experience) are given serious operational consideration at the moment.
So here’s what it boils down to more generally: You don’t know something and you want a way to calculate how to take action despite not knowing that thing. In order for the calculation to work, you have to put a number on what you don’t know. One unfancy way to do this is to say “I’m making up a number.” The fancy way is to say: “I’ve developed a very sophisticated method for putting a number on what we don’t know so we can use our other very sophisticated methods for calculating what action to take.” These sophisticated methods for putting numbers on what is unknown are sometimes billed as methods for dealing with “uncertainty” even when
This appropriation isn’t only happening informally in casual usage. In fact, it happens a lot in highly technical contexts. AI/ML research is one such context, but there are similar dynamics in the usage of “uncertainty” at work in psychology, economics, quantitative finance, policy, and operations management.1 Maybe more troubling: This appropriative use of “uncertainty” gains legitimacy and widespread traction from the reputability of these fields and the people who work in them.
Sure seems like a form of false advertising to me.
Humans need to know things and be certain of them. This need seems visceral, existing mostly below conscious cognition, in the realm of emotion, affect, and instinct. Maybe this is why we have such strong implicit beliefs about certainty and the possibility of being certain. It may also be why we find it so hard to change such beliefs even when faced with evidence that such beliefs are not consistent with reality.
Non-risk not-knowing (which includes true uncertainty) is intensely and viscerally scary because it isn’t quantifiable and cannot be managed away. Overloading and appropriation can both be understood as ways of managing this visceral terror of the unknown. Overloading takes lots of very scary non-risk not-knowing and labels all of it as “risk,” which is quantifiable and manageable. Very comforting. Appropriation takes specific and manageable types of not-knowing and labels them as “uncertainty,” suggesting that true uncertainty is now manageable. Also very comforting.
Both forms of inaccurate usage provide false comfort by creating the impression that unknowns are more manageable and controllable than they actually are. The words we use make us misdiagnose the situations we confront. That’s what’s happening with overloading and appropriation (the diagram at the beginning shows how overloading and appropriation relate to other types of not-knowing).
The words we use to describe a situation of not-knowing shape how we think about acting in that situation. Both overloading and appropriation are bad because they warp how we think about acting when we face different forms of not-knowing.
If we say a situation is “risky” we tend to act using methods that are only appropriate for formal risk situations — this is bad when we use “risk” to refer to non-risk situations. Conversely, if we say a method is good for dealing with “uncertainty” we tend to think we can use it when faced with non-risk situations — this is bad when the method is actually only good for formal risk.
We need to say, as precisely and unambiguously as we can, what we mean when we are talking about the many different types of not-knowing that exist. The challenge is that our words for not-knowing — “risk,” “uncertainty,” “ignorance,” “ambiguity,” etc — are all inadequate. The best word we have for non-risk not-knowing is Knight’s definition, and that isn’t perfect either because
This is why I’ve been using “not-knowing” instead as a blanket term to describe all the situations of partial knowledge we face. “Not-knowing” includes formal risk, true uncertainty, and forms of non-risk not-knowing that aren’t included in Knightian true uncertainty.
From here on I’ll be deep-diving into these different types of non-risk not-knowing — my goal being to understand how each type differs in how you come to know it, relate to it, and deal with it.
For more on this, take a look at item 4 in the discussion highlights from the fifth discussion on not-knowing.↩︎