A new grant from the Laboratory for Analytic Sciences will fund computing work at Kenyon and two other institutions as they develop tools to evaluate systems that use AI.
The $335,281 grant is a collaboration between Professor of Computing Jordan Crouser and researchers at the University of North Carolina at Chapel Hill and Worcester Polytechnic Institute.
“This project brings together a lot of different perspectives, and that’s really exciting,” said Crouser, founding director of Kenyon’s new interdisciplinary program in computing. “There are opportunities to get students involved as developers, as well as to work on the theory behind the metrics we implement.”
This work builds on prior research by Crouser and his collaborators on the development of EvalOps, a system of continuous evaluation throughout a project’s lifecycle rather than waiting until the project’s completion. It is supported by the Laboratory for Analytic Sciences, a Department of Defense-sponsored partnership between academia, industry and government led by North Carolina State University.
This round of funding supports the development of novel tools to help developers of AI-enabled systems used by knowledge workers to measure system performance. These new tools enable evaluation of how these systems affect analyst performance and organizational outcomes throughout the software’s development, implementation and deployment process.
“It’s hard to know if you’ve built a good thing until after you’ve implemented and deployed it, and that’s a really expensive proposition,” Crouser said.
That’s why the instruments being developed at Kenyon and its partner schools are so important, he explained.
“When we think about someone having to make a decision with something that is AI-generated, they need to know if they can trust it,” Crouser said.
In order to develop useful tools for measuring performance, this project will address questions such as: How do we know if a large language model is trustworthy? How do we know if it is efficient? Does it need to be retrained in specific areas in order to get defined performance gains?
And by involving students, the work should benefit from their unique perspectives, Crouser said.
“Undergraduates bring a kind of intellectual fearlessness and curiosity, and that adds unparalleled value; their willingness to think expansively and ask unexpected questions often surfaces insights that experienced researchers overlook,” he said. “As we extend EvalOps to support continuous evaluation of AI-enabled systems, Kenyon students will play a real role in shaping where the work goes next.”
Read more about Crouser and how his students are shaping the future of computing at Kenyon.