Demystified XR
A 4-Dimensional Art Experience: Uncovering the Nameless Women in John Singer Sargent’s Paintings
(Please unmute the video for better viewing experience.)
Independent Work | 2019 Device: Magic Leap, HTC VIVE Skills: XR Design, Unity Development Tools: Unity, Blender, MLTK, VRTK
(Demystified-XR was inspired by a hackathon demo of augmenting arts with the Magic Leap by a team of 5 women, Robin Janz, Cieraej, Lauren Slowik, Annelie Koller, and me.
This page documents my independent work inspired by that demo).
Brief
“Demystified XR” suggests a new way of viewing art-pieces in galleries or museums that is “seductive” rather than indoctrinating.
Viewers are not given full sight of the art-piece at first, instead, they have to clarify the mist around that piece, both literally and figuratively.
Archival materials relevant to different parts of the painting will be triggered by user interaction.
Viewers have to piece them together mentally before the entire painting is unveiled.
Demystified XR hopes to make art-viewing a more involving, constructive experience, and to put art into larger historical contexts.
The Painting
This painting caught our eyes for a few reasons. (1) It is one of those paintings that are noteless but worth digging into.
When researching the woman in the painting, we found her family history and personal stories with Helen Keller attested by archival materials (letters, census data, pictures, and even items still intact today), that renders this woman in the 19th century less distant.
(2)The little details in this painting have implications for big historical events such as the 1893 Chicago World Fair,
which makes the piece “demystifi-able”.
(3) The woman in the painting, Mrs Charles E. Inches, is arguably a woman who climbs the social ladders
with a disregard of rules. As such, she personifies the bold innocence in this painting. Her attitude and her life still inspire women of today.
(4) Last but not least, the painter, John Singer Sargent, is known for painting a series of women.
Sargent’s Women has a distinct elegance and potency.
Who were these women? The stories of these female connect our past with our present and remind us how differently we live now.
To Engage
The average museum goer spends 17 seconds viewing an art piece and rarely has the chance to dive into the mists of the piece.
Meanwhile, there are countless digital archival materials available nowadays that could help unravel the details in art history.
Demystified XR is intended to plunge a viewer into the world of an art piece, hopefully creating longer, educational engagement.
To Recognize
We decided to give those extra viewing seconds to less well-known paintings.
Not every painting enjoys the same acclaim as Van Gogh's starry night does. The neglected ones deserve some attention.
We are particularly interested in paintings that depict women and other historical figures who have been marginalized or overlooked.
In art history, who gets painted is a matter of social status and politics. Thus, discovering the women that were empowered in darker times means a lot to our generation.
To Uncover
Rather than simply animating the painting, “Demystified” gives life to a world of related digital archival materials rarely viewed by the public.
It immerses viewers in the contextual history of an art-piece, inviting them to discover the secrets, and connect the dots.
(Below are some archives involved in this project, including 1893 Chicago World Fair pics, Sargent’s female paintings, and Helen Keller’s letters.)
Experience Design
We were able to gather a list of archival materials around our chosen art piece.
From there we began brainstorming
on interactions with these elements, considering that the experience will likely happen in a museum setting, participants are supposed to interact with the augmented elements in an elegant way from a certain distance.
Initially, we conceived of 4 scenes (the dress scene, the ferries wheel - Chicago
World Fair scene, the painter’s signature scene, and the 1893 Boston scene), all of which are inspired and triggered by various details on the painting, with related archival materials available.
Then we mapped out the behavior of scene objects as well as user interactions through storyboarding.
In summary,
the user follows a four-step task flow at each scene.
1. clearing the mist,
2. trigger the “artifact”,
3. experience the history behind that artifact,
4. (mist returns) continue demystifying.
AR Prototype
We envision this experience to be both VR and AR friendly. The 1st prototype was built with Magic Leap,
primarily using its image and gesture recognition functions.
Imagine “clearing the mists” in front of a painting using your hands in a gallery or museum…
Other viewers who see this will be intrigued to find out what’s happening and probably try engaging with the painting as well.
During user test, we found that the image tracking of Magic Leap wasn’t as high fidelity as we were hoping.
Besides, the luminance level in the room affects image/gesture recognition as well.
VR Development
I further developed the AR prototype in VR.
(1) Designed the game scene, including assets, light effects, post processing effect, volumetric lighting, and particle effects…
(2) Coded VR controller interactions with Unity C# scripts. (3) Designed and coded sound for the whole experience.
(4) Animated 10 assets using the Unity animation system, Blender rigging/shapekey animation with scripts.
Scene1 Development in Unity & Blender
Scene2 Development in Unity
What I learned
“How to make an experience both AR and VR compatible from a design level,
how to iterate XR experiences with different fidelity,
how to make my script reusable in Unity VR development.”
Future Work
1. Site Testing: test the full extent of possible experiences, ideally at the Boston MFA where the painting is currently on view.
2. Build An AR Platform: should application in a real museum setting prove effective, we could create a multi-contributor AR platform
to enable museum curators to create these experiences and users to leverage Creative Commons archives.
3. Interactions: design more sophisticated interactions related to art history or art education, potentially involving other inputs.