Claudio Spiess

PhD Candidate, Computer Science

cvspiess at ucdavis dot edu
DECAL Lab
University of California, Davis
Davis, CA

Bio

Hey there! I’m Claudio 👋🏼 I’m a PhD candidate in Computer Science at the University of California, Davis, affiliated with the DECAL Lab. My research interests are in the intersection of Natural Language Processing and Software Engineering. I study how machine learning techniques, mostly from the NLP world, can understand source code, and to a greater extent, help software engineering. In particular, most of my work is focused on applying LLMs (Large Language Models) to source code. I’m fortunate to be advised by Prof. Prem Devanbu, and to have collaborated with exceptional colleagues around the world. My work has been published at flagship venues such as ICSE and FSE, and I have served as a reviewer for premier journals such as TOSEM.

From model “understanding”, we can do useful things like program generation from natural language, automated bug fixing, automated documentation, anomaly detection (bugs!), reverse engineering, naming, among many others. Not only do I seek to build systems, but to investigate metaphysical questions: do LLMs understand code? What do they learn? What biases and problems do these approaches have? And most importantly, how to fix them? On the practical side, I’m interested in how cognitive load can be alleviated while writing software by smart tools for programmers. Between 2023 and 2025, my main focus was on the calibration of LLMs for code, or rather the lack thereof. In 2025, I also worked on prompt programming languages and automated prompt optimization for LLM agents. Recently, I have been working on how LLMs understand and reason about code.

Previously, I helped build a data driven lending platform at Dutch FinTech startup Floryn as a full stack software engineer, making machine learning work for loans. Most processes had some form of machine learning algorithms backing them, so I built interesting interpretation and explanation tools for non-technicals.

I received my bachelor degree in Computer Science & Engineering from the Free University of Bolzano. I wrote a research thesis, concentrating on NLP for software engineering, under the supervision of Dr. Romain Robbes and Dr. Andrea Janes. During this time, I was affiliated with the Software and Systems Engineering (SwSE) research group.

When I’m not hacking around on code or models, I like to travel the world with a backpack (44 countries/territories and counting), scuba dive (150 dives and counting), and hike volcanoes. I also speak six languages: English, German, French, Dutch, Italian, and some Spanish.

Publications

Claudio Spiess, Prem Devanbu, Earl T. Barr
How Robustly Do LLMs Understand Execution Semantics?

AIware 2026, 2026

We study the robustness of LLM program-output prediction under code transformations and input perturbations, including inputs that raise exceptions. The results expose failures hidden by performance on the original benchmark.

PublicationPreprint Cite

Roham Koohestani, Agnia Sergeyuk, David Gros, Claudio Spiess, Sergey Titov, Premkumar Devanbu, Maliheh Izadi
Does In-IDE Calibration of Large Language Models work at Scale?

FSE 2026, 2026

Collaboration with JetBrains Research Prior work suggests calibration can improve alignment, but at-scale evidence is limited. In this work, we investigate the feasibility of applying calibration of code models to an in-IDE context. We study two aspects of the problem: (1) the technical method for implementing confidence calibration and improving the reliability of code generation models, and (2) the human-centered design principles for effectively communicating reliability signal to developers.

PublicationPreprint Cite

Ali Al-Kaswan, Claudio Spiess, Prem Devanbu, Arie van Deursen, Maliheh Izadi
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs

MSR 2026, 2026

We study how training-data exposure to buggy and fixed code affects code LLM preferences and generations, using membership testing to distinguish memorization from generalization.

Publication Cite

Francisco Ribeiro, Claudio Spiess, Premkumar Devanbu, Sarah Nadi
On LLMs’ Internal Representation of Code Correctness

ICSE 2026, 2026

Inspired by findings that LLMs internally encode concepts like truthfulness, this paper explores if LLMs similarly represent code correctness. Specifically, we identify a correctness representation inside LLMs by contrasting the hidden states between pairs of correct and incorrect code for the same programming tasks. By experimenting on four LLMs, we show that exploiting this extracted correctness representation outperforms standard log-likelihood ranking, as well as verbalized model confidence. Furthermore, we explore how this internal correctness signal can be used to select higher-quality code samples, without requiring test execution.

PublicationPreprint Cite

Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, Toufique Ahmed
Calibration and Correctness of Language Models for Code

ICSE 2025, 2025

Machine learning models are widely used but can also often be wrong. Users would benefit from a reliable indication of whether a given output from a given model should be trusted, so a rational decision can be made whether to use the output or not. In this work, we studied whether current LLMs for code are well calibrated.

PublicationPreprint Cite

Claudio Spiess, Mandana Vaziri, Louis Mandel, Martin Hirzel
AutoPDL: Automatic Prompt Optimization for LLM Agents

AutoML 2025, 2025

Work done at IBM Research This paper proposes AutoPDL, an automated approach to discover good LLM agent configurations. Our method frames this as a structured AutoML problem over a combinatorial space of agentic and non-agentic prompting patterns and demonstrations, using successive halving to efficiently navigate this space. We introduce a library implementing common prompting patterns using the PDL prompt programming language. AutoPDL solutions are human-readable, editable, and executable PDL programs that use this library.

PublicationPreprint Cite

Claudio Spiess
STraceBERT: Source Code Retrieval using Semantic Application Traces

ESEC/FSE 2023, 2023

Novel approach that utilizes a Java dynamic analysis tool to record calls to core Java libraries, and a BERT-style model on the recorded application traces for effective method source code retrieval from a candidate set. Experiments demonstrate the effectiveness in retrieving the source code compared to existing approaches. Proposed approach offers a promising solution to the problem of code retrieval in software reverse engineering and opens up new avenues for further research in this area.

PublicationPreprint Cite

Projects

Mandana Vaziri, Louis Mandel, Claudio Spiess, Martin Hirzel
PDL: A Declarative Prompt Programming Language

arXiv, 2024

Work done at IBM Research This paper introduces the Prompt Declaration Language (PDL). PDL is a simple declarative data-oriented language that puts prompts at the forefront, based on YAML. PDL works well with many LLM platforms and LLMs. It supports writing interactive applications that call LLMs and tools, and makes it easy to implement common use-cases such as chatbots, RAG, or agents.

Preprint Cite

Claudio Spiess, Romain Robbes, Andrea Janes
Method Name Suggestions: An Open Vocabulary Approach

Bachelor Thesis, 2019

Performed a series of experiments evaluating architectural and data preprocessing changes on the accuracy of the names produced by the STOTA code2seq model. We found that BPE can successfully be applied for method naming, resulting in model performance up to 4.2 F1 points higher than baseline

ResearchGate

Claudio Spiess, Baris Sen, David Gayda, Stefan Broeger
Universal Transformer: Towards Learned Positional Encodings

Class Project, 2022

A novel pretraining approach for architecture and modality agnostic Transformer positional embeddings.

GitHub

Claudio Spiess, Stefan Broeger
Impact of War on Food Security in the Middle East & East Africa

Class Project, 2023

Visualized food imports, exports, and balances in the Middle East and East Africa using Tableau

Tableau

Claudio Spiess
Simulating crop yields in El Oro, Ecuador

Class Project, 2023

Created interactive visualizations of GeoTiff files representing actual yields, potential yields, mean temperature, etc for various climate change scenarios using ipyleaflet, geopandas, rasterio, and more.

GitHub

Claudio Spiess, Stefan Broeger
SPM: Social Package Manager

Class Project, 2022

A pip wrapper that uses Distributed Hash Tables (DHT) to store which packages you and your friends have.

GitHub

Claudio Spiess
CUDA Docker Stack

Personal Proect, 2023

Docker image providing CUDA-acclerated Jupyter Lab environment, with PyTorch and HuggingFace.

GitHub

Claudio Spiess
Kickstarter Project Analysis

Personal Project, 2021

Performed basic exploratory data analysis on a public Kickstarter dataset to produce recommendations for a successful Kickstarter project.

GitHub

Claudio Spiess
Jtrak to Macdive

Personal Project, 2020

A small script to convert JTrak dive logs to Macdive format.

GitHub

Claudio Spiess
Perry

Personal Project, 2018

Perry is an asset tracker (GPS) designed to run on an Arduino, with a React powered web interface and Node backend.

GitHub

Claudio Spiess
Bookhub

Class Project, 2017

Book sharing database

GitHub

Claudio Spiess
Address Book

Class Project, 2017

A simple address book written in Java using JavaFX and JUnit for unit testing.

GitHub

Claudio Spiess
Learning English

Class Project, 2017

A quiz application for learning English.

GitHub

Claudio Spiess, François Tronche-Macaire, Riccardo Felluga
Natürliche Rechenmaschine

Class Project, 2019

A simple Yacc/Lex calculator, in German.

GitHub

@inproceedings{spiess2026execution,
  title = {{How Robustly Do LLMs Understand Execution Semantics?}},
  author = {Spiess, Claudio and Devanbu, Prem and Barr, Earl T.},
  year = {2026},
  booktitle = {Proceedings of the 3rd ACM International Conference on AI-Powered Software},
  pages = {288--298},
  publisher = {ACM},
  url = {https://doi.org/10.1145/3805760.3814919},
  doi = {10.1145/3805760.3814919},
}

@inproceedings{koohestani2026calibration,
  title = {{Does In-IDE Calibration of Large Language Models work at Scale?}},
  author = {Koohestani, Roham and Sergeyuk, Agnia and Gros, David and Spiess, Claudio and Titov, Sergey and Devanbu, Premkumar and Izadi, Maliheh},
  year = {2026},
  booktitle = {Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering},
  pages = {609--619},
  publisher = {ACM},
  url = {https://doi.org/10.1145/3803437.3805234},
  doi = {10.1145/3803437.3805234},
}

@inproceedings{alkaswan2026exposure,
  title = {{Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs}},
  author = {Al-Kaswan, Ali and Spiess, Claudio and Devanbu, Prem and van Deursen, Arie and Izadi, Maliheh},
  year = {2026},
  booktitle = {Proceedings of the 23rd International Conference on Mining Software Repositories},
  pages = {86--97},
  publisher = {ACM},
  url = {https://doi.org/10.1145/3793302.3793341},
  doi = {10.1145/3793302.3793341},
}

@inproceedings{ribeiro2026correctness,
  title = {{On LLMs’ Internal Representation of Code Correctness}},
  author = {Ribeiro, Francisco and Spiess, Claudio and Devanbu, Premkumar and Nadi, Sarah},
  year = {2026},
  booktitle = {Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering},
  pages = {1160--1172},
  publisher = {ACM},
  url = {https://doi.org/10.1145/3744916.3787846},
  doi = {10.1145/3744916.3787846},
}

@inproceedings{spiess2025calibration,
  title = {{Calibration and Correctness of Language Models for Code}},
  author = {Spiess, Claudio and Gros, David and Pai, Kunal Suresh and Pradel, Michael and Rabin, Md Rafiqul Islam and Alipour, Amin and Jha, Susmit and Devanbu, Prem and Ahmed, Toufique},
  year = {2025},
  booktitle = {2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE)},
  pages = {540--552},
  publisher = {IEEE},
  url = {https://doi.org/10.1109/ICSE55347.2025.00040},
  doi = {10.1109/ICSE55347.2025.00040},
}

@inproceedings{pmlr-v293-spiess25a,
  title = {{AutoPDL: Automatic Prompt Optimization for LLM Agents}},
  author = {Spiess, Claudio and Vaziri, Mandana and Mandel, Louis and Hirzel, Martin},
  year = {2025},
  booktitle = {Proceedings of the Fourth International Conference on Automated Machine Learning},
  pages = {13/1--20},
  volume = {293},
  series = {Proceedings of Machine Learning Research},
  publisher = {PMLR},
  editor = {Akoglu, Leman and Doerr, Carola and van Rijn, Jan N. and Garnett, Roman and Gardner, Jacob R.},
  month = {September},
  pdf = {https://raw.githubusercontent.com/mlresearch/v293/main/assets/spiess25a/spiess25a.pdf},
  url = {https://proceedings.mlr.press/v293/spiess25a.html},
}

@inproceedings{spiess2023stracebert,
  title = {{STraceBERT: Source Code Retrieval using Semantic Application Traces}},
  author = {Spiess, Claudio},
  year = {2023},
  booktitle = {Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  pages = {2207--2209},
  publisher = {ACM},
  url = {https://doi.org/10.1145/3611643.3617852},
  doi = {10.1145/3611643.3617852},
}

@misc{vaziri2024pdldeclarativepromptprogramming,
  title={PDL: A Declarative Prompt Programming Language},
  author={Mandana Vaziri and Louis Mandel and Claudio Spiess and Martin Hirzel},
  year={2024},
  eprint={2410.19135},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2410.19135},
}