The Transregional Collaborative Research Center (TRR 318) „Constructing Explainability“ investigates how explanations of algorithmic decisions can be jointly constructed by the explainer and the explainee. The project C05 investigates how human decision makers and intelligent systems can collaboratively explore a decision problem to make a decision that is accountable and hence explainable. The goal is to enable medical experts to understand and assess medical decisions and their implications by posing queries and receiving back causal or counterfactual answers from the intelligent system. The project develops the required methods for probabilistic causal reasoning, decision-process analysis, and language-based interaction. We study medical decision-making processes, develop a formal exploration and decision process model, and apply it in an interactive system that guides a medical expert towards a better, more explainable decision.