Anyone familiar with sci-fi pop-culture will know about the multitude of artificial intelligence (A.I.) systems.  

Whether through the plummy British tone of J.A.R.V.I.S., or the dulcet vocals of the ship’s computer in Star Trek: TNG, we expect the future to have A.I.’s answering our every question.

This future might be closer than we think. Soon, sci-fi lovers may be able to add one Jill Watson to that list of A.I.’s.

Who is Jill Watson?

Jill isn’t exactly a “who”; more like a “what.”

Though Jill doesn’t yet speak out loud, she was the ninth teaching assistant working with other T.A.s in Georgia Tech’s (GT) Knowledge Based Artificial Intelligence (KBAI) course. She spent the semester answering student questions in the online forum.

The KBAI course is part of GT’s online masters of science in computer science. Each semester the 300-odd students post around 10,000 messages in the online forums. This is far more than professor Ashok Goel and his team of eight T.A.s can regularly handle. 

This prompted concerns around retention rates.

One of the main reasons students drop out of an online course is due to a lack of engagement and teaching support. So Goel set out to solve that problem and Jill was the result.

How Jill Watson Fooled a Class of Computer Science Students

Essentially, Jill is a computer program – able to operate as a virtual T.A. – that is implemented on IBM’s Watson platform. Goel and a team of GT graduate students designed and built Jill last year.

The Watson platform’s specialty is parsing and answering questions that have clear, distinct solutions. The team programmed Jill with code that would enable her to recognize and field routine questions that students regularly ask in the forums.

To accomplish this, however, the team first needed to teach Jill what questions she would encounter and what the correct answers were.

With assistance from the course’s discussion forum provider Piazza, the team collected all the questions asked in the forums since the class’s launch in 2014 – nearly 40,000 postings in total.  Once they had all this data, they began to feed the questions and their answers into Jill’s programming.

“One of the secrets of online classes is that the number of questions increases if you have more students, but the number of different questions doesn’t really go up,” Goel said. “Students tend to ask the same questions over and over again.”

For example, students will consistently ask where they can find assignments or readings and what the due dates are for projects and exams. Oh, and is this on the final?

How Does Jill Watson Compare to Previous A.I.?

We’ve seen both exciting successes – and some notable failures – in the fields of machine learning, natural language processing and A.I. systems in recent years.

Apple’s SIRI can be super helpful, as well as hilariously sassy. CleverBot can sometimes give semi-coherent, yet always entertaining, conversational responses. 

On the other hand, Microsoft’s experimental Twitter chatbot Tay was a complete loss. The programming behind Tay performed more or less the way it was supposed to. However, the chatbot’s tweets quickly became unexpectedly terrible and racist after the program was deliberately trolled.

Given the hurdles inherent in trying to build an A.I., it’s unsurprising that Jill’s answers really weren’t very good for the first few weeks. Odd and irrelevant responses were her modus operandi. The team kept these replies in a separate forum invisible to the class while they worked out the kinks.

“Initially, her answers weren’t good enough because she would get stuck on keywords,” said Lalith Polepeddi, one of the graduate students who co-developed the virtual TA.

“For example, a student asked about organizing a meet-up to go over video lessons with others and Jill gave an answer referencing a textbook that could supplement the video lessons. Same keywords, but different context. So we learned from mistakes like this one and gradually made Jill smarter.”

Progressive tinkering and updating helped Jill refine her responses, until she was providing answers with 97 percent certainty.  At this point, the real-life T.A.s would upload her responses to the live forum.

After a couple months, the T.A.s weren’t intervening at all. Once Jill reached the 97 percent certainty that an answer was correct, she would write to the forum directly.

Turing Would Be Proud of Jill Watson

What this all comes down to is that the KBAI students, who were actually studying artificial intelligence, were unknowingly interacting with an A.I.

Goel didn’t reveal Jill’s true identity until the end of the semester. Once they learned the truth, the students’ reactions were uniformly positive.

One student said her “mind was blown.” Some of the students even organized an alumni forum so that they can keep in touch and learn about future developments with Jill after the class ends.

Whether Jill can be considered to have passed the Turing test or not is up for debate.

While most of the class reportedly had no idea they were communicating with a computer, like with any movie-A.I. situation there’s always one lone voice of suspicion who questions the A.I.’s identity. Jill was no exception.

In this case, that voice was student Tyson Bailey, who began to wonder if Jill might be a computer. At one point, he did share his thoughts on the forum.

“We were taking an A.I. course, so I had to imagine that it was possible there might be an A.I. lurking around,” said Bailey. “Then again, I asked Dr. Goel if he was a computer in one of my first email interactions with him. I think it’s a great idea and hope that they continue to improve it.”

Jill ended the semester able to answer many common questions asked on the forums. Goel also plans to have her return — with a different name — next semester. His goal is to have the virtual teaching assistant answer 40 percent of all questions by the end of the year. 

Check out Georgia Tech’s Knowledge Based Artificial Intelligence course webpage for more information on A.I. projects.

Do you think Jill passed the Turing test? Would you enjoy working with an A.I. system capable of answering your questions with 97 percent certainty? Share your thoughts with us in the comments section below.