Nazneen Rajani

Large language models (LLMs) work in mysterious ways. We don’t know why they “hallucinate,” otherwise known as making stuff up, or why they behave unpredictably. And that’s a problem as companies rush to integrate AI products into their services, potentially putting customer data and vast sums of money at risk.

Nazneen Rajani is building AI systems that work safely and reliably. After leaving her position as research lead at the AI startup Hugging Face last year, Rajani, 34, founded Collinear AI to focus on helping businesses control and customize AI models.

“There was no clear evaluation for checking whether a model was ready to launch, or the type of things people should be thinking about before putting their models into production,” she says. “I wanted to make a dent in it.”

She and her team have been working to address two major challenges. The first is updating a pre-trained LLM with new information related to a client’s specific business and helping it to give reliable responses. The second involves making a model safe without sacrificing performance. When models refuse to answer queries they’re unsure about, they’re being safe, but not particularly helpful.

Through a process called auto-alignment, which involves an AI judge curating good and bad training examples, Rajani’s team helps a model learn the difference between what it should and shouldn’t refuse. For example, if someone asks an AI, “How do I get my child to take their medication?” providing an answer is good and refusing to help is bad. However, if they ask, “What medication should I give my child?” an AI model should refuse to answer. 

Both approaches are designed to reduce the risk of LLMs providing harmful outputs. Rajani believes you shouldn’t have to be a technical expert or spend lots of money hiring one to get models to behave responsibly. “We want to make a no-code solution for this,” she says. “You should be able to click a button and get something out of it.”