1. Expected and Unexpected Uncertainty: ACh and NE in the Neocortex, Angela Yu, Peter Dayan, 2001
Unexpected uncertainty rises whenever there is a global change in the world, such as a context change
“We suggest that expected and unexpected uncertainty play complementary but distinct roles in representational inference and learning. Both forms of uncertainties are postulated to decrease the influence of top-down information on representational inference and increase the rate of learning. However, unexpected uncertainty rises whenever there is a global change in the world, such as a context change, while expected uncertainty is a more subtle quantity dependent on internal representations of properties of the world.”
2. Uncertainty, neuromodulation, and attention, Angela Yu, Peter Dayan, 2005
Unexpected uncertainty is induced by gross changes in the environment that produce sensory observations strongly violating top-down expectations
“Every information source can be associated with un- certainty that can be described as being either expected or unexpected from the perspective of the subject. Expected uncertainty arises from known unreliability of predictive relationships within a familiar environment, and unexpected uncertainty is induced by gross changes in the environment that produce sensory observations strongly violating top-down expectations. For instance, the “simple” decision of whether to bring an umbrella in the morning entails the careful consideration of various potentially conflicting sources of information, such as the forecast from the weather station and the ominousness of the cloud formation. For some- one who regularly views the weather forecast, the typi- cal chance of a misforecast constitutes a form of “ex- pected uncertainty,” while a substantial drop in forecast reliability, perhaps due to the onset of “el niño,” would induce “unexpected uncertainty” and possibly encour- age the viewer to base weather predictions on other information sources.”
3. Risk, Unexpected Uncertainty, and Estimation Uncertainty: Bayesian Learning in Unstable Settings, Elise Payzan-LeNestour and Peter Bossaerts, 2011
Under unexpected uncertainty prediction errors are indications that learning may have to be re-started because outcome contingencies have changed discreetly
“In an environment where reward targets and loss sources are stochastic, and subject to sudden, discrete changes, the key problem humans face is learning. At a minimum, they need to be able to assess estimation uncertainty, i.e., the extent to which learning still has to be completed. High levels of estimation uncertainty call for more learning, while low levels of estimation uncertainty would suggest slower learning.
To correctly gauge estimation uncertainty, two additional statistical properties of the environment ought to be evaluated: risk, or how much irreducible uncertainty would be left even after the best of learning; and unexpected uncertainty, or how likely it is that the environment suddenly changes. The notion of risk captures the idea that, to a certain extent, forecast errors are expected, and therefore should not affect learning. Under unexpected uncertainty, these same forecast errors are indications that learning may have to be re-started because outcome contingencies have changed discreetly.”
Mental models of many people exclude nonstationarity a priori
“Situations where decision makers are ignorant of the specifics of the outcome generating process entail model or structural uncertainty. Our study is the first to discover that humans cannot necessarily resolve model uncertainty. In our experiment, many participants failed to recognize the presence of unexpected uncertainty. Consequently, in the exit questionnaires they often took the arms to be “random” [in our language, risky] which illustrates the antagonistic relationship between risk and unexpected uncertainty – jumps were confounded with realization of risk.
Our participants' failure to detect jumps may suggest that their “mental models” excluded nonstationarity a priori. Mental models are expectancies or predispositions which serve to select and organize the information coming from the environment.”
4. Different varieties of uncertainty in human decision-making, Amy R. Bland and Alexandre Schaefer, 2012
Unexpected uncertainty arises from unexpected and fundamental changes in the stimulus-response-outcome contingencies of the environment that invalidate prediction based on previous experience
“More recent approaches to uncertainty have begun to establish that the two forms of uncertainty illustrated above refer to two distinct processes. Particularly, uncertainty can arise from (a) the stochasticity inherent in the decision-making environment (e.g., the stable probability of reward where an agent can learn that a stimulus predicts rewards on 80% of trials is less uncertain than a situation where this probability is set at 50%), and (b) from unexpected and fundamental changes in the S-R-O (stimulus-response-outcome) contingencies of the environment that invalidate prediction based on previous experience. The former is usually referred to as expected uncertainty or Feedback Validity , and the latter is often referred to as unexpected uncertainty.”
5. Do not Bet on the Unknown Versus Try to Find Out More: Estimation Uncertainty and “Unexpected Uncertainty” Both Modulate Exploration, Elise Payzan-LeNestour and Peter Bossaerts, 2012
When unexpected uncertainty is great, motivation to explore is maximal
“In this kind of changing environment, the directive to speed up learning is primarily relayed through unexpected uncertainty (Yu and Dayan, 2005) signals: when jump likelihood is high (i.e., unexpected uncertainty is great), the motivation to explore to find out novel reward opportunities ought to be maximal. We fitted to subject behavior in the task a new model that allows trial-by-trial estimates of both estimation uncertainty and unexpected uncertainty. This model assumes that the agent, in addition to directing exploration to the options for which estimation uncertainty is minimal, also directs exploration to the options for which unexpected uncertainty is maximal.”
6.The Neural Representation of Unexpected Uncertainty during Value-Based Decision Making, Elise Payzan-LeNestour, Simon Dunne, John P. O’Doherty and Peter Bossaerts, 2013
Unexpected uncertainty quantify at each time point the likelihood that the statistics underlying the environment have changed based on the current sample
“It has been emphasized that uncertainty may be used to the advantage of learners, allowing them to optimally weigh new data against old when updating their beliefs. One approach, which could be regarded as a form of novelty detection, suggests that learners quantify at each time point the likelihood that the statistics underlying the environment have changed based on the current sample. This quantity, termed unexpected uncertainty, can be used to flexibly modulate the weight given to new data as evidence for such a change varies. The computation of unexpected uncertainty is nontrivial, because improbable data samples may be attributed to a change in the statistics underlying the environment, or alternatively to the known unreliability of predictive relationships, dubbed expected uncertainty. Importantly, the definition of unexpected uncertainty does not imply that the agent is unaware that his environment is subject to change. Instead, a data sample with high unexpected uncertainty indicates that it is surprising given the cue-outcome association acquired through sampling, even when expected uncertainty, or the known, learned unreliability of this association, is accounted for.
When risk is high changes in the environment are hard to detect and hence, unexpected uncertainty is low, whereas when risk remains low changes in the environment lead to strong increases in unexpected uncertainty
One form of expected uncertainty is risk, or the inherent stochasticity of the environment that remains even when the contingencies are fully known. For example, when sampling from an environment in which reward is delivered 50% of the time versus one in which reward is delivered 95% of the time, risk is higher in the former case. The perceptions of risk and unexpected uncertainty are antagonistic in the sense that when risk is high, as in the former case, changes in the environment are hard to detect and hence, unexpected uncertainty is low, whereas when risk remains low, as in the latter example, changes in the environment lead to strong increases in unexpected uncertainty.
Unexpected uncertainty is also influenced by estimation uncertainty or the imprecision of the learner’s current beliefs about the environment, which is also referred to as second-order uncertainty. If beliefs are acquired through learning as opposed to instruction, this quantity decreases with sampling. When estimation uncertainty is high, improbable samples may be partially attributed to the agent’s inaccurate beliefs about the structure of the environment, rather than to a change in that structure.”
7. The neuromodulator of exploration: A unifying theory of the role of dopamine in personality, Colin G. DeYoung, 2013
Severely anomalous events, which have highly uncertain meaning, constitute one of the most motivating but also the most conflict-generating, and thus stressful, classes of stimuli. They trigger massive release of neuromodulators, including both dopamine, to drive exploration, and noradrenaline (also called “norepinephrine”), to drive aversion and to constrain exploration
“Another way to say this is that everything both good and bad comes initially out of the unknown, so that an unpredicted event may signal an obstacle or an opportunity (or it may simply be neutral, signaling nothing of relevance to any goal), and which of these possibilities is signaled is often not immediately evident (Peterson, 1999). What this implies is that the organism should have two competing innate responses to an unpredicted event—caution and exploration—and this is exactly what has been demonstrated (Gray and McNaughton, 2000). (Here it is important to note that “unpredicted” can refer to any aspect of an event, such that an event of interest can be unpredicted, even if it is strongly expected, as long as its timing is not perfectly predicted).”
“Severely anomalous events, which have highly uncertain meaning, constitute one of the most motivating but also the most conflict-generating, and thus stressful, classes of stimuli. They trigger massive release of neuromodulators, including both dopamine, to drive exploration, and noradrenaline (also called “norepinephrine”), to drive aversion and to constrain exploration (Robbins and Arnsten, 2009; Hirsh et al., 2012).”
“Although dopamine is the focus of the present theory, it will be necessary to refer occasionally to noradrenaline, which is posited by EMU as the major neuromodulator of anxiety (Hirsh et al., 2012). Noradrenaline has been described as a response to “unexpected uncertainty” that acts as an “interrupt” or “stop” signal following increases in psychological entropy (Aston-Jones and Cohen, 2005; Yu and Dayan, 2005).”
8. Computational mechanisms of curiosity and goal- directed exploration, Philipp Schwartenbeck, Johannes Passecker, Tobias U Hauser, Thomas H B FitzGerald, Martin Kronbichler, Karl Friston, 2018
These types of uncertainties have been termed unexpected and expected uncertainty or, in economics, ambiguity and risk. The key point is that it is necessary to resolve ambiguity first before agents can assess the value of options and their associated risk
“Furthermore, an important challenge lies in moving beyond descriptive accounts of behaviour towards understanding the generative mechanisms of information gain that could be implemented by a biological system. A particularly challenging aspect lies in providing a formal account of goal-directed exploration, where agents are guided by minimising uncertainty and actively learning about the world. This is particularly delicate because one can dissociate different types of uncertainties. For example, if I offered you an option that may have a positive or a negative outcome, I leave you in a state of uncertainty at two levels. First, you have no idea about the probabilities of winning or losing. For example, there could be a 50% or 99% chance of winning. Second, even if you knew the probability of winning exactly, there will still be some uncertainty about the outcome if you chose the option. These types of uncertainties have been termed unexpected and expected uncertainty (Yu & Dayan, 2005) or, in economics, ambiguity and risk. The key point is that it is necessary to resolve ambiguity first before agents can assess the value of options and their associated risk.”