Chris Callison-Burch: Research Statement

(Last updated November 2023)

In the field of Natural Language Processing, the rise of Large-Language Models (LLMs) like ChatGPT represents a historical milestone towards fulfilling the longstanding objective of developing AI systems that are capable of understanding and generating natural language. Although ChatGPT seems like an overnight success, its history can be traced to the 1990s and the introduction of statistical n-gram language models, and to the introduction of neural LMs in the early 2000s. In the mid-2010s, we witnessed the rise of pre-trained models like word2vec, ELMo, and BERT, which redefined the state-of-the-art by employing vector space representations and the transformer architecture to capture the nuances of natural language semantics. Then, we entered the current generation of Generative Pretrained Transformer (GPT) models like ChatGPT, Llama, and PALM, which, powered by immense training data and huge computational power, have ushered in an entirely new era for the NLP community.

These advances have caused a paradigm shift in our field. Like many other NLP professors, I initially struggled to understand how I could continue to make a research contribution. On numerous occasions, I have related how my using GPT-3 for the first time in early 2021 lead to what I describe as a “career existential crisis” because it seemingly supplanted many NLP tasks that were previously first order research concerns. I realized that research areas like machine translation, automatic summarization, and question answering had largely been subsumed by GPT-style models. Moreover, because these models require Google-sized data centers to train, I believed that no university would ever be able to create one.

What makes the advent of LLMs profoundly transformative is their ability to act as general-purpose learners. The paradigm of “pre-train then fine-tune” gives them the flexibility to adapt to new tasks, while requiring a fraction of the training data that was previously required. Their proficiency in few-shot learning and instruction following have outstripped most previous machine learning methodologies. In essence, we now have general purpose AI systems that can observe and learn from a handful of examples, and also interpret and execute fairly complex instructions articulated in natural language. This means that NLP is now delivering tools that are genuinely transformative –not only to our own field– but to society as a whole.

Is there still room for academics like me to make a research contribution in the era of large pretrained models? I have come to the position that there is still a place for academic research in NLP. Over the past two years, I’ve figured out several research directions that are a good fit for academic researchers in this era. My research directions are now:

1. Understanding the capabilities of LLMs
2. Building explainable models
3. Enabling symbolic reasoning with LLMs
4. Building LLM-based applications
5. Advising policymakers on AI and the law

Understanding the Capabilities of Large Language Models

I aim to understand the capabilities and the limitations of LLMs through my research. I have explored a variety of different aspects of LLMs including prompting techniques, model architectures, synthetic datasets, data preprocessing, and human detectability of machine generated text.

Prompting Techniques An important attribute of LLMs is their ability to adapt to new tasks given a relatively small number of examples. This characteristic, termed as “few-shot learning,” effectively demonstrates how these models perform in-context learning which recognizes patterns within a prompt rather than requiring weight updates. In our work on prompting techniques, we have leveraged this trait to push the boundaries of what large language models can accomplish.

In Reif et al. (ACL 2022), we demonstrated the flexibility of language models in replicating novel styles beyond conventional style transfer tasks through a technique called augmented zero-shot learning. This approach frames style transfer as a sentence rewriting task and requires only a natural language instruction, without any model fine-tuning or exemplars in the target style. Through human evaluation, we showed that this method can achieve strong performance not just on standard style transfer tasks like sentiment, but also on non-standard natural language transformations such as making text more melodramatic or inserting metaphors.

In Zhang et al., (NLRSE 2023), we explored the emerging crossover of training language models on a mixture of text and code, which has been shown to improve performance on structured reasoning tasks. We conducted a systematic comparison of code vs. text prompts across diverse tasks like QA, summarization, and sentiment analysis. Our key finding was that while code prompting boosts performance on certain reasoning tasks, text prompts remain more effective for most NLP tasks. We also analyzed the impact of code prompt style and instruction fine-tuning. This work sheds light on this rapidly growing area at the intersection of code and language.

Model Architectures My collaborators and I have studied various architectures for LLMs. In Ippolito et al. (NAACL 2022), we explored using the same underlying model architecture for both continuation-based and fill-in-the-blank text generation tasks. We found that models pre-trained with a fill-in-the-blank objective were effective at both tasks, while models pre-trained for continuation struggled with insertion, necessitating separate models for each task. Our work highlighted the viability of a single model able to handle this variety of related text editing operations.

In Patel et al. (ICLR 2023)), we compared unidirectional versus bidirectional architectures and their ability to perform few-shot learning. While unidirectional models like GPT-3 have shown impressive few-shot abilities, bidirectional models often require task-specific fine-tuning. We introduced a prompting technique enabling bidirectional models to achieve strong few-shot performance without fine-tuning. Surprisingly, our prompted bidirectional models outperformed unidirectional models on tasks like machine translation despite using fewer parameters.

Synthetic Datasets LLMs also have utility in producing synthetic datasets. In an effort to better combat biases present in naturally occurring data, we used LLMs to create a high-quality synthetic dataset for generating Wikipedia-style biographies from info-boxes (Yuan et al., NeurIPS 2021). Specifically, we had the LLM generate draft attribute lists and biographies describing fictional individuals, which were then revised by human annotators. This allowed us to construct a dataset with more control over gender, nationality, and other attributes compared to using real Wikipedia data, enabling better evaluation of models’ abilities to generate grounded, faithful text. Our work provides a case study for how LLMs can be incorporated into a human-AI collaborative pipeline to efficiently curate datasets with desired properties. This line of inquiry indicates a promising prospect of LLMs being used to generate data for machine learning tasks beyond language processing. Overall, judiciously leveraging LLMs’ generative capabilities could enable creation of higher-quality datasets that better reflect the true complexity of target tasks and distributions. In Patel et al (EMNLP 2023) we use LLMs to create a synthetic stylometry dataset by prompting GPT-3 to describe the styles of 1000 authors, and then train classifiers to reliably predict stylistic signatures of new authors.

Data Preprocessing In Lee et al. (ACL 2022), we demonstrated that de-duplicating training data for LLMs is an incredibly important step. This seemingly minor adjustment in data preprocessing pipeline has major effects. By keeping only a single copy of sentence in the training data, the frequency of ‘memorization’ in LLMs significantly reduced. This has implications for whether model outputs may potentially infringe on copyright, and our de-duplication technique substantially mitigates this risk. As a result, our proposed methodology has now become a standard preprocessing technique for training LLMs/

Detecting Machine Generated Text My research group has also explored how well people can distinguish “real” human-written text from “fake” machine-generated text. In a large-scale human study analyzing the boundary between human-written and machine-generated passages, we analyzed how well participants could distinguish machine-generated text from human-written counterparts (Dugan et al., AAAI 2023). We found that while humans struggled at this task – and that it’s getting harder for people to detect machine generated text as model sizes increase. We also found there was substantial variance in annotator skill. With proper incentives and training, annotators improved over time. This work demonstrated the viability of training humans to better detect the boundary between human and machine text.

We also explored the automatic detection of machine-generated text (Ippolito et al., ACL 2020). We found that automatic detection of generated text is easiest when humans are fooled. Our experiments highlighted that certain settings in decoding methods like nucleus sampling could be optimized for fooling humans, they introduce statistical anomalies like over-representation of common tokens that classifiers found easy to detect. This work revealed the promise of using both human evaluation and automatic detectors to fully assess generation systems, since they focus on different and complimentary aspects of the problem.

Each of these lines of inquiry adds to a broader understanding of LLMs’ capabilities, forms a significant portion of my ongoing research, and will continue to guide my research initiatives over the coming years.

Building Explainable Models

Explainability is the extent to which why a model makes certain predictions can be presented in understandable terms to some target audience. Faithfulness is a criterion for explainability. A faithful explanation should accurately reflect the reasoning process behind the model’s prediction. In Lyu et al., 2022 (in submission to Computational Linguistics), we survey nearly 100 publications related to explainability in NLP, and discuss whether they are faithful.

Concept Bottleneck Models As deep learning systems continue to advance, there is a growing need to make them more interpretable and explainable, especially as they are deployed in high-stakes domains like healthcare, finance, and transportation. However, the black box nature of neural networks makes explainability challenging. My group has been exploring concept bottleneck models as one approach to address this problem and make large language models (LLMs) more explainable while maintaining or improving accuracy. The key intuition is to force the model to predict an intermediate conceptual representation that provides a human-understandable bottleneck grounding the model’s predictions. For instance, in image classification we can force the model to first predict concepts like “black line running through yellow face”, “black horn on each side of their head” and then use these in a simple linear model to predict a bird species like “horned lark”. The concepts give a much better window into the model’s reasoning than an end-to-end black box neural model. In collaboration with Prof. Mark Yatskar and our jointly supervised PhD students, we published a CVPR paper (Yang et al., CVPR 2023) that demonstrated that LLMs could be used to generate concept bottlenecks to produce more explainable image classifiers.

Faithful Chain-of-Thought Models In Lyu et al. (IJCNLP-AACL 2023), we introduce a new method for generating inherently faithful model explanation. Our “faithful chain-of-thought” approach builds on the chain-of-thought prompting technique. Chain-of-Thought prompting is a few-shot learning technique that demonstrates for LLMs how to “reason aloud” and has demonstrated improvements in performance on tasks like solving math word problems. However, there’s no guarantee that the final answer will be faithful to the reasoning chain. With Prof. Eric Wong and several PhD students here at Penn, we introduced a method for using an LLM to translate a chain-of-thought reasoning chain into from natural language into a symbolic language (like Python, Datalog or PDDL) and then executing the symbolic reasoning chain with a solver. This guarantees that the output answer will match the reasoning chain, thus ensuring faithfulness. Moreover, we showed our method sets new state-of-the-art performance on 7 diverse reasoning tasks including math word problems and multi-hop question answering. Our paper won the Area Chair Award for the best paper in the “Interpretability and Analysis of Models for NLP” area at IJCNLP-AACL 2023. Our work was done concurrently with several similar efforts like “Program of Thought”.

Enabling Symbolic Reasoning with LLMs

LLMs have proven weak at symbolic reasoning, but in our faithful chain-of-thought work we showed that LLMs can be used to generate structured representations, like executable Python code, which are passed to a solver. The solver output is used to guide the LLM to iteratively refine its structured prediction. This allows combining the reasoning capabilities of symbolic systems with the natural language prowess of LLMs.

Reasoning about Events via Goal-Step Inference I participated in the DARPA Knowledge-directed Artificial Intelligence Reasoning Over Schemas (KAIROS) program and used LLMs throughout the program to produce structured representations. Early in the program my research group used using BERT to perform several new reasoning tasks that are relevant for reasoning about events. In Lyu et al. (EMNLP 2020), We used data that we mined from the website WikiHow to create 2 challenge tasks: (1) reasoning about goal-step relations, and (2) reasoning about the temporal ordering of steps. We created a multiple-choice dataset through intelligent sampling of WikiHow.

In task 1, a system is given a goal and 4 candidate steps and needs to choose the step that helps achieve the goal. For example, given the goal “Prevent Coronavirus” and the candidate steps:

1. wash your hands
2. wash your cat
3. clap your hands
4. eat your protein

where the correct step would be A. In task 2, given a goal and 2 steps, a system needs to determine which step temporally precedes the other. For example, given the goal “Clean Silver” and the steps:

1. dry the silver
2. handwash the silver

where the correct answer would be “B precedes A”.

We showed that fine-tuning BERT to be able to perform these tasks improved its abilities on other benchmarks like the commonsense reasoning benchmark SWAG, the intent detection benchmark Snips, and the Story Cloze narrative reconstruction task. In particular, after fine-tuning to our dataset, higher accuracy could be achieved on all of those with fewer of the within-task training examples.

Goal-Oriented Script Construction In Lyu et al. (INLG 2021), we explored using LLMs to generate steps towards a goal in natural language. We fine-tuned multilingual T5 on WikiHow to generate scripts in several languages. We contrasted this LLM generation-centric approach to a retrieve-then-reorder approach, and we designed careful human evaluations. We explored this for 18 languages. We used the retrieve-then-reorder approach to create scripts that connected into the KAIROS Event Ontology.

Visual Goal-Step Inference In Yang et al. (EMNLP 2021), we adapted our goal-step reasoning tasks to include visual inputs. For example, given the text goal “Bake Fish” we would have the system pick which image represented a step towards that goal, again creating a set of confusable images that were highly visually similar. We found that pre-training on our visual dataset allowed us to do better transfer to other tasks, in this case few-shot performance on COIN and on Howto100m, which are both video datasets related to how-to instructions.

Hierarchical Goal-Step Relationships In Zhou et al (ACL 2022), we examined the hierarchical relationships between goals and steps, where steps in a procedure can be recursively designated as goals themselves, allowing us to construct a rich nested structure of sub-events. We defined a model for linking steps onto goals, trained it on WikiHow, and evaluated the resulting hierarchical procedures on the task of video retrieval from Howto100M.

Causal Reasoning about Entities and Events In Zhang et al (EACL 2023), we introduced a more complex causal reasoning benchmark dataset. In this dataset we combine entity state tracking (e.g., the temperature of a pan) and counterfactual event reasoning (e.g., how likely am I to burn myself by touching the pan), by tracking entities and attributes across tasks. In this causal reasoning paper, we also examined the use of code-based LLMs to do entity state tracking.

Zero-shot and Few-shot Schema Generation In two ACL papers this year, Li et al (ACL 2023) and Zhang et al (ACL 2023), we examined the potential for LLMs to directly generate schema using zero-shot and few-shot prompting, optionally with humans in the loop. This shows the promise of using LLMs to create structured representations like the KAIROS Schema Graph, which can then be used with external symbolic reasoners.

Building LLM-based Applications

In addition to analyzing the capabilities of LLMs and enabling new functionality via symbolic reasoning, I have also explored new applications of LLMs:

Intelligent tutoring systems I have a “Smart Textbook” team of undergraduate + graduate researchers that has developed tools and techniques to create educational applications using large language models. In Dugan et al. (ACL 2022), we conducted a study analyzing the feasibility of generating quiz questions for a course using LLMs. We find that they can produce high-quality flashcard-style quiz questions from textbook chapters a substantial portion of the time, and that the rate improves if textbook chapters are first automatically summarized. The question quality still lags behind human-written summaries. Building on this work, I collected hundreds of student-written summaries from students in my AI class, finding that there is high variance across questions derived human-written summaries. Gonzalez et al. (BEA 2023) introduced a method for enhancing human-written summaries with LLMs that improves question generation quality by rewriting bullet-point style summaries into full sentences.

Building on this Smart Textbook work, two undergraduates from my group created an startup called Martian with an Ed Tech platform called “Learn” (Upadhyay et al. EACL 2023). Learn is an end-to-end platform for generating, selecting, and iteratively improving question banks using LLMs. Learn uses prompt engineering and reinforcement learning from human feedback to generate questions tailored to a course’s preferred style. It also employs spaced repetition and item response theory to personalize quiz questions based on individual students’ proficiency. In experiments with 1000 students, we found Learn boosted exam scores and was preferred to traditional methods.

In other papers published at the Building Educational Applications workshop, we showed that augmenting educational videos with automatically generated summaries (Gonzalez et al. BEA 2023), and we analyzed new ways for LLMs to generate interactive coding challenges (Upadhyay et al. BEA 2023) and mathematical exercises as part of this goal.

I aim to build on these Smart Textbook projects to create a full-featured intelligent tutoring system for use in my own courses.

Improving smart home assistants In joint work with Prof. Andrew Head, a new HCI professor here in UPenn’s Department of Computer and Information Science, we are investigating ways to enhance the capabilities of voice assistants like Amazon Alexa and Google Home. In Huang et al (DIS 2023), my PhD student Alyssa Huang, who I co-advise with Prof. Head, conducted an observational study of users interacting with voice assistants for complex tasks like following cooking recipes. The study revealed several deficiencies including information overload, fragmentation, and poor question answering. Based on these insights, Alyssa proposed techniques for the voice assistant to “rewrite the script” in order to adapt written instructions into a more conversational form better suited for audio interaction. We aim to implement some of these capabilities by leveraging LLMs to perform summarization, splitting instructions into substeps, reordering time-sensitive information, and answering questions by reasoning about events. Our interdisciplinary collaboration combines expertise in human-computer interaction and natural language processing to uncover deficiencies in current voice assistants and explore ways to augment them with LLMs.

Language-based games like Dungeons & Dragons While I was a part-time visiting researcher at Google in 2019 and 2020, I proposed that they explore Dungeons & Dragons as a challenge problem for dialogue systems built on large language models. Artificial Intelligence has a long and rich history of using games as challenge problems that lead to advances in the field. In many cases, AI game-playing systems have gone on to rival human champions of the game. Language-based creative games like D&D are a great challenge for LLMs, since they exercise abilities like

• Language generation (multi-party dialog, generating descriptions of the world/player actions, and storytelling)
• Language understanding (knowledge acquisition and representation, state tracking, automated reasoning)
• Planning / strategic play during battles (similar to chess or go)

My group has published several papers exploring how large language models can be used to for D&D. Callison-Burch et al. (EMNLP 2022)describes the work I did at Google framing D&D as a dialogue challenge and train LLMs. My research group has published several follow-on papers. In Zhu et al. (ACL 2023), we construct a dataset called FIREBALL containing structured game state information from thousands of D&D gameplay sessions, measuring how well LLMs can generate grounded narration, and be trained to perform state-tracking. In Zhu et al. (AAID 2023), we develop CALYPSO, a system of LLM-powered interfaces to provide DMs with information, narration, and brainstorming assistance. CALYPSO was used by players in live games for months. Finally, Zhou et al. (ACL 2023) explores modeling theory of mind for the DM to anticipate player actions. Together, this body of work shows the potential for LLMs to enhance creative tabletop gaming as NPCs, automated gamemasters, and co-creative assistants. I’m using this work to teaching students about LLMs in my graduate seminar Interactive Fiction and Text Generation.

Python Framework for Building LLM Applications We recently published an open source Python framework for developing LLM applications called Kani (Zhu et al, NLP-OSS 2023) Kani is a lightweight, flexible, and model-agnostic open-source framework for building language model applications. Kani helps developers implement a variety of complex features by supporting the core building blocks of chat interaction: model interfacing, chat management, and robust function calling. All Kani core functions are easily overridable and well documented to empower developers to customize functionality for their own needs. Kani thus serves as a useful tool for researchers, hobbyists, and industry professionals alike to accelerate their development while retaining interoperability and fine-grained control. Kani has nearly 500 stars on Github. I’ve been teaching a PhD seminar on the intersection of LLMs and Programming Languages with Benjamin Pierce, a PL faculty member here at UPenn, and I hope to continue working on research at the intersection of PL and LLMs in the future.

Advising Policymakers on AI and the Law

I was on sabbatical when ChatGPT was announced, and I spent much of my time trying to do good scientific communication to much broader audiences than I typically speak to. I gave 20 invited talks and panels to a wide range of groups. During my sabbatical, I spent time to research legal issues related to AI and copyright that arise from LLMs’ ability to generate creative content. I served on the US Copyright Office’s AI listening tour. In May, I was invited to testify before The U.S. House of Representatives. I traveled to Washington DC to appear before the House Judiciary’s Subcommittee on Courts, Intellectual Property, and the Internet, which held a hearing on “Artificial Intelligence and Intellectual Property: Interoperability of AI and Copyright Law”. I explained generative AI to lawmakers and advocated that the training of AI systems on copyrighted materials should be considered fair use.

I have become an expert at issues related to copyright law and AI. I participated in the GenLaw workshop at ICML 2023 and made many connections with legal scholars who are focused on this issue. Lawyers from companies like Meta, OpenAI and Microsoft have reached out to me about potentially serving as an expert witness in pending lawsuits that may shape the future of generative AI. I have been actively trying to engage with policy makers by submitting comments to the US Copyright Office, reaching out to lawmakers who are drafting legislation on the right to publicity, and engaging in advocacy for fair use. I have also attempted to build bridges to communities who are worried about AI misappropriating their creative work – by speaking to groups like The Society of Composers and Lyricists – because I think mutual understanding is critical to navigating future policies for AI. I am also planning to explore new research directions aimed at developing techniques to mitigate copyright risks for generative models.

Selected Talks

• The U.S. House of Representatives Judiciary Committee, Subcommittee on Courts, Intellectual Property, and the Internet. Testimony at hearing on "Artificial Intelligence and Intellectual Property: Part I – Interoperability of AI and Copyright Law". May 17, 2023
• Duolingo. Using Large Language Models to Generate Course Materials. March 16, 2023
• Penn ASSET Seminar. Using Large Language Models to Build Explainable Classifiers. February 8, 2023
• Penn Medicine Seminar. Ask An Expert About ChatGPT. February 14, 2023

Works Cited

@inproceedings{lee2022deduplicating,
  title={Deduplicating Training Data Makes Language Models Better},
  author={Lee, Katherine and Ippolito, Daphne and Nystrom, Andrew and Zhang, Chiyuan and Eck, Douglas and Callison-Burch, Chris and Carlini, Nicholas},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)},
  address={Dublin, Ireland}
  year={2022}
}

           

@inproceedings{Reif2022-style-transfer,
  title={A Recipe For Arbitrary Text Style Transfer with Large Language Models},
  author={Emily Reif and Daphne Ippolito and Ann Yuan and Andy Coenen and Chris Callison-Burch and Jason Wei},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)},
  address={Dublin, Ireland}
  year={2022}
}

           

@inproceedings{Ippolito-2022-fill-in-the-blank,
  title={The Case for a Single Model that can Both Generate Continuations and Fill in the Blank},
  author={Daphne Ippolito and Liam Dugan and Emily Reif and Ann Yuan and Andy Coenen and Chris Callison-Burch},
  booktitle={Findings of the 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistic (NAACL 2022)},
  address={Seattle, Washington}
  year={2022}
}

           

@inproceedings{Patel-ICLR-2023,
  url = {https://arxiv.org/abs/2209.14500},
  author = {Patel, Ajay and Li, Bryan and Rasooli, Mohammad Sadegh and Constant, Noah and Raffel, Colin and Callison-Burch, Chris},
  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Bidirectional Language Models Are Also Few-shot Learners},
  booktitle={Eleventh International Conference on Learning Representations (ICLR 2023)},
  address={Kigali, Rwanda}
  year={2023}
}

           

@inproceedings{Yuan2021SynthBio,
  title={{SynthBio}: A Case Study in Human-{AI} Collaborative Curation of Text Datasets},
  author={Ann Yuan and Daphne Ippolito and Vitaly Nikolaev and Chris Callison-Burch and Andy Coenen and Sebastian Gehrmann},
  booktitle={35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks.},
  year={2021}
}

           

@inproceedings{patel2023learning,
      title={Learning Interpretable Style Embeddings via Prompting LLMs}, 
      author={Ajay Patel and Delip Rao and Ansh Kothary and Kathleen McKeown and Chris Callison-Burch},
      booktitle={Findings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023 Findings)},
      address={Singapore}
      year = {2023}
}

           

@inproceedings{zhang-etal-2023-exploring,
  title={Exploring the Curious Case of Code Prompts},
  author={Li Zhang and Liam Dugan and Hainiu Xu and Chris Callison-Burch},
  booktitle={Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)},
  address={Toronto, Canada},
  year={2023}
}

           

@inproceedings{dugan-ippolito-et-al-2023,
  title={Real or Fake Text? Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text},
  author={Liam Dugan and Daphne Ippolito and Arun Kirubarajan and Sherry Shi and Chris Callison-Burch},
  booktitle={The 37th AAAI Conference on Artificial Intelligence (AAAI 2023)},
  address={Washington DC, USA}
  year={2023}
}

           

@inproceedings{Ippolito-Duckworth-et-al:2020,
 author = {Daphne Ippolito and Daniel Duckworth and Douglas Eck and Chris Callison-Burch},
 title = {Automatic Detection of Generated Text is Easiest when Humans are Fooled},
 booktitle = {Proceedings of The 58th Annual Meeting of the Association for Computational Linguistics (ACL)},
 year = {2020},
 url = {http://www.cis.upenn.edu/~ccb/publications/automatic-detection-of-generated-text-is-easiest-when-humans-are-fooled.pdf}
} 

           

@article{Lyu-et-al:CL:2024,
   author    = {Qing Lyu and Marianna Apidianaki and Chris Callison-Burch},
   title =   {Towards Faithful Model Explanation in {NLP}: A Survey},
   journal = {Computational Linguistics},
   year =    {2024},
   volume = {50},
   number = {2},
   pages = {657–723},
   url = {https://doi.org/10.1162/coli_a_00511}
 }

           

@inproceedings{yang-etal-2023-language-in-a-bottle,
  title = {Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification},
  author = {Yang, Yue and Panagopoulou, Artemis and Zhou, Shenghao and Jin, Daniel and Callison-Burch, Chris and Yatskar, Mark},
  booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023)},
  year = {2023},
  address = {Vancouver, Canada},
  publisher = "IEEE/CVF",
  url = {https://www.cis.upenn.edu/~ccb/publications/language-in-a-bottle.pdf}
}

           

@inproceedings{lyu2023faithful,
    author = {Lyu, Qing and Havaldar, Shreya and Stein, Adam and Zhang, Li and Rao, Delip and Wong, Eric and Apidianaki, Marianna and Callison-Burch, Chris},
    title = {Faithful Chain-of-Thought Reasoning},
    booktitle = {Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023)},
    year = {2023},
    location = {Bali, Indonesia},
    date = {November 1--4}
}

           

@inproceedings{lyu-zhang-wikihow:2020,
 author = {Qing Lyu and Li Zhang and Chris Callison-Burch},
 title = {Reasoning about Goals, Steps, and Temporal Ordering with WikiHow},
 booktitle = {Proceedings of The 2020 Conference on Empirical Methods In Natural Language Proceedings (EMNLP)},
 year = {2020},
 url = {http://www.cis.upenn.edu/~ccb/publications/reasoning-about-goals-with-wikihow.pdf}
} 

           

@inproceedings{Lyu-et-al:2021,
 author = {Qing Lyu and Li Zhang and Chris Callison-Burch},
 title = {Goal-Oriented Script Construction},
 booktitle = {Proceedings of the 14th International Conference on Natural Language Generation (INLG 2021)},
 year = {2021},
 url = {http://www.cis.upenn.edu/~ccb/publications/goal-oriented-script-construction.pdf}
}

           

@inproceedings{yang2021visual,
  title={Visual Goal-Step Inference using {wikiHow},
  author={Yang, Yue and Panagopoulou, Artemis and Lyu, Qing and Zhang, Li and Yatskar, Mark and Callison-Burch, Chris},
  booktitle={Proceedings of The 2021 Conference on Empirical Methods In Natural Language Proceedings (EMNLP)},
  year={2021},
  url={http://www.cis.upenn.edu/~ccb/publications/visual-goal-step-inference-using-wikihow.pdf}
}

           

@inproceedings{Zhou-Zhang-et-al-2022-wikihow-hierarchies,
  title={Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data},
  author={Shuyan Zhou and Li Zhang and Yue Yang and Qing Lyu and Pengcheng Yin and Chris Callison-Burch and Graham Neubig},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)},
  address={Dublin, Ireland}
  year={2022}
}

           

@inproceedings{zhang-etal-2023-causal,
  title = "Causal Reasoning About Entities and Events in Procedural Texts",
  author = "Li Zhang and Hainiu Xu and Yue Yang and Shuyan Zhou and Weiqiu You and Manni Arora and Chris Callison-Burch"
  booktitle = "Findings of the Association for Computational Linguistics: EACL 2023",
  year = "2023",
  address = "Dubrovnik, Croatia",
  publisher = "Association for Computational Linguistics",
  url = "https://arxiv.org/pdf/2301.10896.pdf"
  abstract = "Entities and events have long been regarded as the crux of machine reasoning. Procedural texts have received increasing attention due to the dynamic nature of involved entities and events. Existing work has focused either on entity state tracking (e.g., the temperature of a pan) or on counterfactual event reasoning (e.g., how likely am I to burn myself by touching the pan), while these two tasks are tightly intertwined. In this work, we propose CREPE, the first benchmark on causal reasoning about event plausibility based on entity states. We experiment with strong large language models and show that most models, including GPT3, perform close to chance at .30 F1, lagging far behind the human performance of .87 F1. Inspired by the finding that structured representations such as programming languages benefit event reasoning as a prompt to code language models such as Codex, we creatively inject the causal relations between entities and events through intermediate variables and boost the performance to .67 to .72 F1. Our proposed event representation not only allows for knowledge injection but also marks the first successful attempt of chain-of-thought reasoning with code language models."
}

           

@inproceedings{sha-et-al-2023-creating-event-schema-with-llms,
  author = {Li, Sha and  Zhao, Ruining and Li, Manling and Ji, Heng and Callison-Burch, Chris and Han, Jiawei},
  title = {Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification},
  booktitle={Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023)},
  address={Toronto, Canada}
  year = {2023},  
  copyright = {Creative Commons Attribution 4.0 International}
}

           

@inproceedings{human-in-the-loop-schema-induction,
  author = {Zhang, Tianyi and Tham, Isaac and Hou, Zhaoyi and Ren, Jiaxuan and Zhou, Liyang and Xu, Hainiu and Zhang, Li and Martin, Lara J. and Dror, Rotem and Li, Sha and Ji, Heng and Palmer, Martha and Brown, Susan and Suchocki, Reece and Callison-Burch, Chris},
  keywords = {Human-Computer Interaction (cs.HC), Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Human-in-the-Loop Schema Induction},
  booktitle={Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL 2023 demos)},
  address={Toronto, Canada},
  year = {2023}
}

           

@inproceedings{Dugan-et-al-2022-feasibility-study,
  title={A Feasibility Study of Answer-Unaware Question Generation for Education},
  author={Liam Dugan and Eleni Miltsakaki and Etan Ginsberg and Shriyash Upadhyay and Hannah Gonzalez and Dahyeon Choi and Chuning Yuan and Chris Callison-Burch},
  booktitle={Findings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)},
  address={Dublin, Ireland}
  year={2022}
}

           

@inproceedings{gonzalez-etal-2023-enhancing,
  title={Enhancing Human Summaries for Question-Answer Generation in Education},
  author={Hannah Gonzalez and Liam Dugan and Eleni Miltsakaki and Zhiqi Cui and Jiaxuan Ren and Bryan Li and Shriyash Upadhyay and Etan Ginsberg and Chris Callison-Burch},
  booktitle={Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  address={Toronto, Canada},
  year={2023}
}

           

@inproceedings{upadhyay-etal-2023-learn,
    title = "Learn With Martian: A Tool For Creating Assignments That Can Write And Re-Write Themselves",
    author = "Upadhyay, Shriyash  and
      Callison-Burch, Chris  and
      Ginsberg, Etan",
    booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.eacl-demo.30",
    pages = "267--276",
    abstract = "In this paper, we propose Learn, a unified, easy-to-use tool to apply question generation and selection in classrooms. The tool lets instructors and TAs create assignments that can write and re-write themselves. Given existing course materials, for example a reference textbook, Learn can generate questions, select the highest quality questions, show the questions to students, adapt question difficulty to student knowledge, and generate new questions based on how effectively old questions help students learn. The modular, composable nature of the tools for handling each sub-task allow instructors to use only the parts of the tool necessary to the course, allowing for integration in a large number of courses with varied teaching styles. We also report on the adoption of the tool in classes at the University of Pennsylvania with over 1000 students. Learn is publicly released at https://learn.withmartian.com.",
}

           

@inproceedings{gonzalez-etal-2023-automatically,
  title={Automatically Generated Summaries of Video Lectures May Enhance Students’ Learning Experience},
  author={Hannah Gonzalez and Jiening Li and Helen Jin and Jiaxuan Ren and Hongyu Zhang and Ayotomiwa Akinyele and Adrian Wang and Eleni Miltsakaki and Ryan Baker and Chris Callison-Burch},
  booktitle={Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  address={Toronto, Canada},
  year={2023}
}

           

@inproceedings{upadhyay-etal-2023-improving,
  title={Improving Mathematics Tutoring With A Code Scratchpad},
  author={Shriyash Upadhyay and Etan Ginsberg and Chris Callison-Burch},
  booktitle={Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  address={Toronto, Canada},
  year={2023}
}

           

@inproceedings{10.1145/3563657.3596059,
author = {Hwang, Alyssa and Oza, Natasha and Callison-Burch, Chris and Head, Andrew},
title = {Rewriting the Script: Adapting Text Instructions for Voice Interaction},
year = {2023},
isbn = {9781450398930},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3563657.3596059},
doi = {10.1145/3563657.3596059},
booktitle = {Proceedings of the 2023 ACM Designing Interactive Systems Conference},
pages = {2233–2248},
numpages = {16},
keywords = {splitting, complex task guidance, remixing, reordering, instructions, voice assistants, voice user interfaces, summarization},
location = {Pittsburgh, PA, USA},
series = {DIS '23}
}

           

@inproceedings{callison-burch-tomar-et-al-2022,
  title={Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence},
  author={Chris Callison-Burch and Gaurav Singh Tomar and Lara Martin and Daphne Ippolito and Suma Bailis and David Reitter},
  booktitle={The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022)},
  address={Abu Dhabi, UAE}
  year={2022}
}

           

@inproceedings{zhu-et-al-2023-fireball-dataset,
  author = {Zhu, Andrew and Aggarwal, Karmanya and Feng, Alexander and Martin, Lara and Callison-Burch, Chris and Choi, Yejin and Ammanabrolu, Prithviraj},
  title = {FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information},
  booktitle={Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023)},
  address={Toronto, Canada}
  year = {2023},  
  copyright = {Creative Commons Attribution 4.0 International}
}

           

@inproceedings{zhu2023calypso,
   title={{CALYPSO}: {LLMs} as Dungeon Masters' Assistants},
   author={Zhu, Andrew and Martin, Lara J. and Head, Andrew and Callison-Burch, Chris},
   booktitle={The 19th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2023)},
   year={2023}
}

           

@inproceedings{Pei-et-al-2023-dnd-theory-of-mind,
  author = {Zhou, Pei and Zhu, Andrew and Hu, Jennifer and Pujara, Jay and Ren, Xiang and Callison-Burch, Chris and Choi, Yejin and Ammanabrolu, Prithviraj},
  keywords = {Computation and Language (cs.CL), Artificial Intelligence (cs.AI), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {An AI Dungeon Master's Guide: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons},
  booktitle={Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023)},
  address={Toronto, Canada}
  year = {2023},  
  copyright = {Creative Commons Attribution 4.0 International}
}

           

@inproceedings{zhu2023kani,
      title={Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications}, 
      author={Andrew Zhu and Liam Dugan and Alyssa Hwang and Chris Callison-Burch},
      year={2023},
      booktitle={3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS)},
      address={Singapore}
      year = {2023}
}

           

@misc{CallisonBurch_2023,
  author = {Callison-Burch, Chris},
  title = {Understanding Generative Artificial Intelligence and Its Relationship to Copyright},
  howpublished = {Testimony before The U.S. House of Representatives Judiciary Committee, Subcommittee on Courts, Intellectual Property, and the Internet},
  month = {May},
  year = {2023},
  note = {Hearing on Artificial Intelligence and Intellectual Property: Part I – Interoperability of AI and Copyright Law},
  institution = {University of Pennsylvania, School of Engineering and Applied Sciences, Department of Computer and Information Science}
}