AI Institute for Societal Decision Making Meeting

These are the notes that I took during the AI Institute for Societal Decision Making Meeting in September 2024.

As with my past blog posts on conferences that have included my notes, the comments and descriptions are my own interpretation of the presentations given by the researchers below and may not necessarily reflect the thoughts or opinions of those researchers. I have tried to add some relevant links to recent research by these presenters to give more context to their work.

Complementarity for Decision-Making

Michael Lee : Cognitive modeling of adaptation and self regulation.

Adaptation within partnerships, typically focuses on how AI adapts to humans but people do also adapt to AI. Part of this partnership requires the prediction of behavior, predictive models of cognition and decision making. Most models of cognition focus on learning but less about adaptation and self regulation.

Lee and Dry task varied the knowledge and confidence of AI recommendations in a search space. People are most confident in the decisions they make when the recommendations are given with about 70% confidence and decrease with higher claimed confidence. This is an example where adaptation doesn’t correspond to just observed facts.

Used a sequential sampling model to predict the decision making based on past experience. Self regulation in SSMs determines where the thresholds of making the decision after gathering evidence.

Question: What if the distributions also change as the thresholds are changing, could those be distinguished experimentally.

Based on observations of decision making (but you can update it without making an error), we can self regulate by changing the boundaries. Shows historiesis. Starts off with wide boundaries of threshold that require a lot of evidence but quickly adapts as you gain experience.

Question from Gretchen: Would this require a different model in H-H teaming? Answer: Unsure to know but interested to learn more about the AI sub-field and how they think of the differences being made.

Question from Aarti: How does this connect to the AI conceptualization of confidence which takes confidence to be part of the decision that is made. Answer: In humans we typically think of confidence not as being generated simultaneously to the decision being made but after it.

Ido Erev: How to adjust environments to facilitate teaming between humans and AI

Laws are typically defined to prevent problematic decision processes, like texting while driving. Policing is self-motivated, which impacts the application of law enforcement. There can be alternatives to enforcement besides policing. Popular example is speed cameras, but those might not be used because everyone drives faster than the limit and they can actually increase danger by making people break before the camera.

Regulations impact the ability to automatically enforce laws, e.g police need to be able to hand out fines when it happens. Also law enforcement can have an incentive to ask for payments instead of fines. This also relates to scientific publishing in wanting to reject papers that don’t cite our own work.

Evaluating scientific papers with choice prediction competition solution. Related to the parking lot paradox showed that CCTV is better at reducing car related crimes compared to violence, perhaps due to planning car related crimes more compared to violence. Edo’s explanation to this is that stealing crimes has a higher probability of being reported to the police compared to the amount of violence crimes that go unreported. Alternative could be an AI system that turns on blue lights when it sees a knife on the screen or gives an attention cue to the human controller.

This approach could improve privacy because you don’t need to keep a record of fights before they happen, only the AI system needs to view them. Moves towards a system of preventing crime instead of punishing individuals.

Question: Could this lead to a minority report situation where crimes are being predicted before they happen, thought crime policing, 1984, etc. If the AI model incorrectly classifies you as intending to commit a crime.

Sudeep Bhatia : Choices between gambles.

Decision making from description vs experience. Risky choice doesn’t look like real decision making, and it is difficult to apply these models onto real world choice because we lack the representation that goes into real world choice. Information processing revolution conceptualized the computer as an analogue for the mind and led them to ask the question of how the mind processes information, and what heuristics it uses. Simon talked about mental processes and mechanisms and used the metaphor of the psychological labratory.

Information representation revolution (2010s-2020s) LLMs are being used to learn representations about the world (is this true?). This raises the question of how the mind represents information rather than how it processes information. Exploring variability in risk taking with large language models Used natural language descriptions of real world risky choices,

Had GPT and humans generate reasons for and against doing the thing that was described in the choice problem. There was a correlation between the ratings between subjects and models individual-specific ratings. Can use cognitive models that rely on representations built from generative AI models.

Question: Since GPT is trained on a large corpus of documents to predict the next word in a sentence, is it necessarily true that it is representing a world model? E.g GPT-4 may represent a string of text as 3k floating point numbers.

Beau Schelble: Getting on the same page information sharing team cognition trust and ethics in human ai teams.

What is the machine and systems perspective of the intersection between humans and AI. Applications in simulations of an emergency response scenario called neocities. First study compared shared mental model similarity in humans and AI found no correlation between the two, but AI teammates reduces perceived shared knowledge and trust in the AI Teammate. AI represents departure from the norm of teaming.

Question: Were people aware that they had an AI teammate?

Second study on examining the role of spatial information in team cognition for human ai teams. Demonstrated the difference between the needs of HHT and HATs. Mental model similarity showed an interaction effect where the HATs that showed information about what the AI was doing was helpful, whereas with human teammates this information was distracting.

Question: What is the explanation for the interaction effect?

Modeling information needs to support team cognition in human AI teams, the perceived shared mental model of the AI teammate had no effect but increasing their knowledge access did.

Comparison of different methods of information sharing (verbal and a different one). Both types of information sharing improves situational awareness and the human perception of the AI shared mental model.

Ethical decision making and the teaming scenario during the briefing and debriefing of Human AI teaming, which isn’t focused much on in the literature. Apologies and denials are ineffective in regaining trust after it is lost. Requires a shared ethical identity for human and AI teams that informs decision making. Concepts like team cognition need to be reassessed within the context of human AI teams, and the existing theory is not set up to handle it.

Question: what are some of the examples of these issues and how can they potentially be addressed.

Group Discussion:

Similar themes between the talks include the idea of having AI that are highly aware of the interactions that they are engaged in with humans. What are some of the principals that should be accounted for when HATs are considered in terms of representations and shared mental models.

Question from Hoda to Sudeep, how do we assess the reliability of LLM models in predicting human-like behavior. Answer: I am suggesting comparing the reasoning processes of LLMs in generating reasons for and against the reasons to do specific tasks.

Ido: Historical examples of decision making from rational choice are not useful for understanding real world decision making but they haven’t led to mainstream applications. They lack prediction validity.

Michael: Seems to be a large difference in how information is represented, there seems to be a qualitatively different way that AI and humans are representing information. The goal of AI shouldn’t be to make AI have representations that are similar to humans.

Beau: Asking LLMs to create associations between parts of tasks are likely to result in a high degree of variance. Lack of structure behind the answers that are being given. Important to develop that structure from the technical side in understanding how structure (world models?) is important.

Sudeep: Not trying to suggest that LLMs have the structured representations or the reasoning processes, and we shouldn’t ask them directly how humans are making decisions. But they have the ability to give some insight into the types of things that people are thinking about when they are deliberating.

Autonomous and Cognitive Models of Decision-Making

Kate Donahue : Human-AI collaboration: role of modeling and models of interaction

Why is it that humans using AI algorithms results in more sub-optimal accuracy compared to performance in the lab. This can be better understood by thinking of behavior of humans and algorithms separately, using a mathematical model that describes relative accuracy. Then we develop a theory of how humans and algorithms interact.

Algorithm narrows down many items to a smaller set. Example is driving with google maps which gives you a few options to choose from out of a large possibility of routes. When are two lists better than one? How likely is it for complementarity choices to result in poor ultimate choice selection.

Consider humans and machines as noisy rankers that have some probability distribution over difference preference rankings. Can consider the human as being bandwidth limited in determining the number of options to show them.

Compare the situation where the human or the machine is more accurate than the other. In this case the accuracy of the human is more relevant for the ultimate performance. Failures to achieve complementarity are more likely when the algorithm is better than the human, which is more often the case in the real world.

Considering chatting with humans and chatbots as a Stackelberg. potentially misaligned rewards. Most challenging case of this communication is when there are disagreements about which options are similar and which are dissimilar.

Other future research: different access to information between humans and AI.

Question: Is it safe to conceptualize GPT as having a reward function for outcomes about things like going to a restaurant when it is really trained to just predict the next word and provide replies that are aligned with humans.

Christian Lebiere: Cognitive Models for Human-AI Teaming

One approach to HATs is to have AI mimic human decision making, maybe the average human approximately or personalize models to your specific teammate. Another approach is to try and correct biases and constraints in human cognition, trying to do this is difficult and requires a really good theory of mind for the human, not just heuristics. Another approach is to just give predictions that are as good as possible.

Bridging the gap between cognitive mechanisms and computational intelligence Strength of these models is their constraints and their theory of memory that is used to generate expectations. Predict things out of the box without any human data.

Considering cases of inconsistencies and variations in human decision making. AI systems can be built to take into account the recommendation of higher order decision making, and then the human determines whether or not they will use the information. Humans overstate the reliability of the recommendations even when they don’t wait for and use those recommendations.

In competitive settings people need to learn progressively to approach the nash equilibrium, so this is applicable even in strictly competitive settings. Deceptive signaling in cyber defense results in a high number of people attacking nearly all the time and a long distribution of people who attack less. This is the result of differences in experience and luck, lucky participants at the beginning continue to attack.

Modeling individual participants using model tracing, makes the model have the same experiences as the human and improve the similarity of predicting the behavior of individuals. This allows you to close the loop and change behavior actively based on the experience of the individual to optimize the deception in cybersecurity tasks.

Mark Steyvers : Communicating uncertainty in human-ai teaming

How do LLMs communicate knowledge uncertainty to humans and how to humans interpret uncertainty in verbal statements. Settings where LLMs are either correct or incorrect, either knows or doesn’t know the answer to a multiple choice question, and needs to communicate that uncertainty to the human.

A lot of research shows that LLMs can’t distinguish between cases where they know and don’t know information. We can encourage an understanding of uncertainty, but the main issue is that that information does not end up in the information that the human is viewing.

Using token likelihoods to assess the relative likelihood of each of the options to the multiple choice question. When you do this the model confidence and accuracy is relatively well calibrated, however humans have a poor understanding of whether or not the LLM is correct.

Question: What do you do with the likelihood of any other non a-d response from the LLM.

Question: How is the human assessing the likelihood of the LLM? Do they read some response of the LLM. How do we know that the certainty when the LLM is answering the question is similar to what it is when it is giving a human an explanation for the question.

Adjusting the length of the explanation leads to people increasing the perceived uncertainty. We can also change it by checking to see if the model has confidence above a threshold and then replying with an explanation that matches the true confidence.

Question: Is there really a single conceptualization of confidence of the LLM models.

Solution is to use RLHF to rank replies based on the confidence of the explanation that matches the true confidence, instead of the experience of Chat-GPT in RLHF which was to prefer answers with high confidence.

Question: From an alignment standpoint do we want the LLM to still be replying when it isn’t sure? Are explanations that sound about the level of the confidence appropriate or should it just never answer if it doesn’t know, and refuse to give an answer.

Jeff Schneider: Uncertainty quantification in autonomous decision making.

Autonomous driving requires uncertainty quantification to determine how to fallback from one system of control to another one. Main system is end-to-end deep learning with behavior cloning, which falls back to rules based, which falls back to a rote decision to pull over.

Considering conditions where multiple failures occur at the same time, and this can happen when there is a lot of correlation between the errors of prediction as well as confidence assessment. This can be accounted for by considering how the errors that might happen would be correlated and thus impact the assessment of confidence and error.

Error predictions from PNNs are normally uncorrelated and thus optimistic to the real world, and this can be fixed with a method called SPNNs which allow for temporally correlated error predictions that are less overconfidence.

Group Discussion: Chair: Paul Lehner

Question: Does this apply to experiment generation? Answer: yes and there are projects currently being worked on for generating experiments.

Question: Does this apply to open ended questions with uncertainty. Answer: Yes and there is a method that can try to calibrate more open response type question typically by reformulating the input so that the LLM is only replying with ‘yes’ or no’

Question: AI trying to optimize for human preferences and humans have a mental model of the AI, is there any theory of how this is working? Answer: The process of how people think about AI is diverse and changing quickly as the public perception of AI changes. AI has high initial trust and then people adapt to experience with it. This is related to XAI since real life explanations are an interactive process of communication and not just an info dump of all the related information to choices being made.

Question: How relevant is explainability and trust in the autonomous driving setting. For autonomous driving trust through experience is more important than understanding, similar to automated flying.

Question: Models calibrating their confidence may result in worse performance from a HAT hypothetically. Answer: In high safety settings then these calibrated confidence ratings are important, like in medical decision making.

Question from Hoda: isn’t the aircraft system very different from airplanes because the trust comes from engineers that know how the aircraft works. Answer: I think that it is no longer the case that engineers understand the aircraft controls and this high trust in rigorous testing is really moving towards a more statistical understanding of trust.

Question for Mark from me: isn’t it possible that this conceptualization of ‘confidence’ when the model is answering the ABCD prompt output is highly different from the confidence when it is generating a long response, but is that a problem?

Influencing Decision-Making Behavior

Fei Fang: Disincentivizing behavior in strategic decision making

Computational game theory. People reviewing the other papers in the same conference, typically done by bidding on the papers you want to review. Potential for collusion with reviewing papers of your friends or rejecting the papers that are similar to yours. Apply game theory to mitigate this issue. In the past this was done by using a similarity score between the papers and maximizing the similarity. Solution is to add noise by setting some probably to the reviewer being assigned to a paper based on similarity. Further improvement by perturbing the optimization to adjust it towards the uniform randomness.

Planning patrol of poachers using machine learning to try and predict the best methods of poachers. Issue is that poachers learn the behavior of the patrollers and change their strategy. Determines schedule based on the constraints of the patrol. Community engagement in anti poaching, having the members serve as informants so the patrollers can define their strategy. Another method is to use non professional community members to supplement the anti-poaching efforts.

Gloria Washington

Culturally relevant artificial intelligence research. AAVE biases in conversational AI. How can we increase public trust, improve conversational AI and inform these about racial concerns. Collecting 800 hours of spoken AAVE, used to influence decision making behavior. Developing a benchmark using this data to determine the performance of voice assistants. Protecting black people against scam robocalls and other deepfake scams. Developing guidelines for the development of fair usage.

Biases in medical diagnoses, schizophrenia diagnoses are often incorrect for black men and women. Models of microagression that can be used on virtual reality platforms to practice what they are saying with black and brown patients. Increases the levels of self reported empathy in participants. Also had a community involvement in understanding the issues associated with medical scenarios.

Manish Raghavan

Relationships between human behavior and AI. Interpreting AI models built from human data and improving their design. Don’t want to interpret behavior as revealed preference. Humans consistently behave differently than their preferences. Example of this is online content consumption that are optimized for watch times and clicks, and many of these are impulsive decisions. Objective functions are being designed to optimize things that have a lot of datapoints but are not well considered decisions.

Building content delivery that is more concerned with the decision making of humans. Need to request preferences against content shows that the maximization of these metrics is not perfect. End to end models trained to maximize clicks inherently assume the preferences of humans without any model of that human. It would be better to build a separate user model since there is often mismatch between the algorithms preferences and the true user preferences. Behavioral modeling that are based on a formal model of human behavior.

Risk assessment calculators in medical domains often do not rely on individual participant data that the doctor has available to. Doctor intuition is typically based on experience with the patient. Can doctors reliably differentiate between patients in cases where the algorithm cannot differentiate between them, yes.

Question: Is there a concern that having the ability to maximize say long term use of an app (years instead of minutes), then you are just delaying the regret and that it may actually be much worse.

Hirokazu Shirado

Social implications of human-ai collaboration in network interactions. Considers a human-ai as a single agent and study the interactions between these agents. Example is a human and AI driving at the same time, chicken game example where you need to have one of the people swerve out of the way. One shot situation doesn’t have an opportunity to benefit, but with repeated reciprocity you can have one person swerve on the first trial and then the other swerve on the next. Example is letting people into a zipper merge onto the highway.

Semi-autonomous driving to serve and prevent a crash. In the chicken game scenario with safety assistance then the cost of both cars going straight is potentially less. Running this online shows that having both cars with auto steering assistant means that the probability of bilateral turn is much higher. There also is no alternating reciprocity. Intelligence assisted driving made human decision making more self centered.

Group Decision: Chair: Bryan Wilder

Question: Modelling from chicken’s game and prisoner’s dilemma revealed transfer from the equilibrium games even when models were trained on one they were able to extrapolate to the other. Also highly dependent on the mental model of the other, where knowing about the payoffs determines the likelihood of cooperating. Answer: There are other conditions of the experiment where they turn on or off the automatic turning to see how that would change the performance finding a one directional effect, turning on the safety assistant quickly moved towards selfish, but in the opposite when starting with the intelligent system and shutting it down meant they didn’t gain trust.

Questions: Can we use off the shelf approaches to addressing these issues or do we need to build new theories. Answer: Gloria: we would need to start from new theories for the things she is concerned with since one big aspect of a specific use case like robo-scams is how emotions like ego impact decision making. Answer from Manish: Need to consider the fact that decisions made in the moment vs. long afterwards are different and collecting different data is a good idea.

Trust, Metrics, and Evaluations

Terri Adams-Fuller

Distrust in artificial intelligence, communication tools for developing trust in AI based on public perception. What are the differences between people who trust AI and those that don’t, assuming that some people have an assumption that things will turn out well, especially in the elderly and younger groups.

Use of AI in decision making for disaster response is more polarizing compared to general AI use. Major concerns are jobs, privacy, and reducing intelligence. Postgrads had the highest level of optimism while no HS had the lowest.

Some differences in the perception of trust in AI in emergency compared to typical use cases of AI. Generally more optimistic people are more likely to trust the use of AI in emergency situations. Exposing people more to general use could make them more likely to support emergency support use cases of AI.

Dan Goldstein : Augmenting human cognition and decision making with AI

Different levels of AI may either deskill people over time, give a temporary boost, or improve your skills. The specific types of tasks being done can impact the way the skills of the user are changed through use of AI tools. Presentation of information also changes how skills are impacted, showing exact actions as they are performed (hurting skills) vs showing a high level plan of actions (helping skills).

Adding additional information like unit conversions can help a question turn into a teachable moment by showing how to do something. Faster decision making for a car based on some preferences. People are unlikely to know that LLMs are making a mistake.

Solution could be to convey confidence intervals, difficult to get compared to answers for quiz questions but they can be estimated. Experimentally they can label numbers as being high and low confidence, which improves accuracy on the task. Big challenge to get calibrated confidences, nothing really works. Also a question of how the people will process the probabilities that are given with the confidences.

Erin Chiou: Meaningful Human Control Recent Advancements in Trust Assessment

Unified theory of acceptance and use of technology. Difficult to assess things like attitude since it is not a belief, but it affects things like trust. Informational and relational paradigms for information processing, reasoning and actions.

Framework of trust that is built on relationships, semiotics, don’t want to focus too much on the informational processing since there are relationships between concepts. Scoping is important to consider when applying a team science and systems engineering approach to AI trust since we don’t want to presume accountability should be given to machines.

Applying relational trust models to measuring responsivity, which will be applied onto a tool that can be used by non-experts to determine the trust levels of their systems prior to design and evaluation of systems. Ability, Human centered, collaboration, contextualization, and prioritization.

Group Discussion Chair: Aarti Singh

Questions: People often display a lack of trust but this doesn’t necessarily impact their decision making, how do we factor this into the analyses of trust. Answer: It is interesting why people often do and don’t trust other people and this does change their behavior, so it’s a question of why this occurs and how it applies to humans. People do have concerns even if it doesn’t impact their behavior so they may be negatively impacted by a lack of trust regardless of changing experiences.

Questions: What are the benefits of designing better collaborators vs. accepting limitations and changing the design of user interfaces or the ways that humans interact with AI systems. Answer: It depends also on what the applications are, and system design gaps that are not accounted for must be filled in by people in real life. Trusting only AI isn’t the only concern but the whole process of developing these tools.