Microsoft mentors two Northwestern women in emerging tech field

Two women with Northwestern University connections served as project leads in a mentor and spotlight program that celebrates and supports diverse creators in mixed reality.

Mixed reality, sometimes called hybrid reality, blends the physical and digital worlds to produce new environments where physical and holographic objects co-exist and interact in real time.

The Windows Mixed Reality initiative is a partnership between Microsoft and SH//FT, an immersive media company that supports diverse voices in the virtual reality and augmented reality industry. Twelve women internationally joined this program that mentored participants’ existing concepts and applications for the new field.

Elizabeth Hunter, who is pursuing a Ph.D. in theater and drama in Northwestern’s School of Communication, is developing a game based on an ancient Greek tragedy for Microsoft’s mixed reality headset, Microsoft HoloLens.

Sofya Akhmametyeva, who earned a master of science in robotics from the McCormick School of Engineering in 2016, is incorporating mixed reality and Microsoft HoloLens technology into artificial intelligence systems to enhance robotic capabilities.

Both women teamed up with Microsoft mentors during the summer to begin work. On Nov. 7 and 8, they visited Microsoft headquarters in Redmond, Washington, along with the other project leads engaged in the initiative.

The event served the dual purpose of furthering Hunter’s and Akhmametyeva’s technological skillsets and developing a support network as they continue work in the field of technology.

Mixed reality: the next phase of digital evolution

Mixed reality incorporates elements from two fields: virtual reality and augmented reality.

Virtual reality relies on headsets to immerse users in a video environment; users cannot see their surroundings or interact with holographic objects in the physical world.

There are no how-to videos on YouTube for this yet. You just have to start doing it and learn as you go.”

Augmented reality uses phone cameras and screens to layer digital, interactive objects on top of the physical environment, but the user does not have the ability to manipulate those objects.

Mixed reality takes this concept one step further, bringing digital elements into the physical environment without using a screen as an intermediary. Holograms, which are digital projections with 3D dimensions, fill the physical environment of the user. They look just like any other object in the user’s surroundings, but instead of being made of physical matter, holograms are made entirely of light. They move and change based on interactions with the physical environment and in response to the user’s motions. When wearing the Microsoft Hololens, multiple users can view and manipulate the same hologram at the same time even if they are in different places.

Ancient Greek tragedy enters the digital age

Hunter is using the technology to bring the audience into the ancient story of “Agamemnon.” She is the founder of Fabula(b), a theater and computational design lab housed at The Garage, Northwestern’s hub for innovation. The lab uses emerging interactive technology to inspire and engage new audiences in classic literature.

Hunter currently is developing a game called “Bitter Wind.” Based on the plot of Aeshylus’s Greek tragedy “Agamemnon,” the game is an immersive and interactive storytelling experience.

“Players have to use the clues provided in the game to figure out which character they are supposed to be portraying and why they want to kill Agamemnon,” Hunter said. “There’s a holographic character who leads players through the scenes and gives them hints. It allows players to become part of the story in a dynamic way.”

“Bitter Wind” involves a series of 3D-printed puzzle pieces that interact with the digital projections to unlock a series of holograms that explain the ancient story.

Hunter previously developed a video game called “Something Wicked,” based on Shakespeare’s “Macbeth.” Hunter credits the interdisciplinary learning environment afforded by academia for her creative approach to teaching the classics.

“Innovation in academia is not about sales,” she said. “It’s about pushing the boundaries of art and technology.”

Hunter received a Center for Interdisciplinary Research in the Arts (CIRA) Grant to develop “Bitter Wind,” and a Segal Design Fellowship to prototype “Something Wicked.”She also took advantage of the knowledge bank available to her at Northwestern. 

“Elizabeth is a bridge builder,” said Harvey Young, Hunter’s Ph.D. advisor and chair of the theater department in Northwestern’s School of Communication. “She understands how the humanities can have a broader impact, not only for those who are passionate about the arts but also those who love design and technology.”

Robots become more human

Akhmametyeva, co-founder of Augmented Intelligence Resource, LLC (AIR), is developing communication channels between robots’ artificial intelligence systems and the Microsoft Hololens, so that the headset can interpret the wearer’s words and gestures and send the signal directly to the robot.

“Mixed reality technology will make communication with robots more intuitive,” Akhmametyeva said. “A human will be able to look at or point to a glass of water and tell the robot, ‘Bring me that glass,’ and the robot will know what to do and how to get there.”

Akhmametyeva, who started AIR with support from The Garage, said the opportunity to network with Microsoft engineers is invaluable to her development of software that allows humans to interface more naturally with robots and vice versa.

"A human will be able to look at or point to a glass of water and tell the robot, ‘Bring me that glass,’ and the robot will know what to do and how to get there."

“We want robots to be as autonomous as possible, and we want humans to be able to easily override the robots’ decisions,” Akhmametyeva said. “Mixed reality makes that possible without touching a keyboard or pushing buttons. You just have to speak or point.”

Akhmametyeva and Hunter said the summit at Microsoft headquarters was a profound opportunity to network with other women in a male-dominated field and to develop close relationships with the engineers at the heart of the emerging technology.

“We accomplished six weeks’ worth of work in two days because when we ran into problems, we just asked the engineer who built the software. We didn’t waste time looking up answers online,” Hunter said, adding that the solutions often can’t be found online because the technology is so new.

“There are no how-to videos on YouTube for this yet,” she said. “There’s an experiential element of the technology that can’t be explained or researched. You just have to start doing it and learn as you go.”