An immersive data visualization that gives users an unconventional perspective on the numbers
- Unreal Engine 4
- VCV Rack
This project was turned in as an assignment in my Data Art class at NYU's Interactive Telecommunications Program (ITP) in the fall of 2018.
It is a data visualization of the global land-ocan temperature index from 1880 to 2017, using the Lowess (5-year) smoothing method, rather than the annual mean.
I wanted to make a visualization that existed outside the bounds of the conventional, two-dimensional line chart or histogram that is scanned and evaluated in a more analytical or passive manner. But I also didn't want to abandon the kind of steep incline that such a visualization (line chart or histogram) might convey on the paper. In short, I wanted to make a creative visualization that was nevertheless impactful, even one that could only be considered creative insofar as it succeeds in its impact in an imaginative or unintuitive way (prioritizing prettiness, satisfying animations, etc., would only have been distracting or detracting from my aim here).
I decided to build something in a three-dimensional, virtual environment. I wanted to engage the user's body and allow them to read the incline of the numbers with an organ other than their eyes.
Climbing in VR is an impactful experience, and standing atop a tall perch and looking down in VR affords an experience similar to what one might expect in the real world: a truer appreciation of the heights reached. (I also included various mechanics for those unable to make the climb, such as elevators and teleporation.)
I ultimately built out the visualization in Unreal Engine 4, featuring navigable meshes procedurally generated from the data, and a motion controller pawn (fancy phrase for the virtual character users inhabit and control) with a locomotion system that allows climbing, allowing users to climb the data. A quick aside on procedural generation... Procedural generation might scan as a bit of an intimidating phrase, but at its core it's really quite simple. All it means is: making a thing according to certain rules. This entails a process whereby I'm not making any one individual thing directly, I'm merely putting forth the rules that for that thing to be made. And once the rules are set, I can stipulate for any number of those things to be made. Sometimes the rules are very broad. In these instances, the things created from the rules can vary. With rules like these, we can create random variations on a loose template of rules (a hundred slightly different mazes, for example). Other times, such as is the case in this very project, the rules are strict: I'm forming the rules such that only a single object will be made, one dictated entirely by the climate change data that I'm piping in.
The programming was chiefly done in UE4's blueprint system, which is a node-based, visual coding environment. The following image contains the blueprint code for the procedurally generated meshes. It might look complicated, but it's essentially just a few nested for loops (highly common all across programming) with various data relating to vector, size, mesh instance and material. The following image gives perspective on the setup of the scene. On the left is the mirror object, in which the climbing user can see themselves, and see how high they've climbed up the visualization (the bar they are currently standing on lights up to indicate their progress).
Here is video documentation of an early version of the experience in action.
My process for the project was as follows:
- Source the climate change data from a reliable organization, and then take this data (a JSON file) and scale it using some of the helpful functions D3.js provides. This scaling allows us to scale up the temperature numbers (though keeping the year data consistent), allowing them to be of better use as values for procedurally generating meshes in Unreal Engine 4, while also maintaining accuracy (not disorting the numbers for exaggerated impact).
- Export the scaled JSON from the D3 script and bring it into Unreal Engine 4 as a data table.
- Create a data struct in UE4 with variables that correspond to the keys in the JSON data, allowing us to work with the numbers in an organized fashion.
- Create an actor blueprint. In this blueprint's construction script, we procedurally create a series of meshes using the climate change data as instructions. This process involves nested forEachLoops as well various nodes relating to math, vector, location, and scale. We have to organize all this code in the correct sequence before we can call the mesh generating nodes, which take all this math, vector, location, and scale data as instruction for creating meshes, which will essentially be the three-dimensional histogram that we are building.
- Create a climbing locomotion system that works across the BP_MotionControllerPawn and BP_MotionController blueprints, using casting as well as blueprint interfaces to communicate between each other as well as with the climable objects (BP_Climable). This system chiefly relies on the getting and setting of “climable” variables, as well as the storing of playing locotion and grip in order to calculate world position and move the pawn based on where the motion controllers pull it to (climbing, essentially).
- Modify collision channels as well as tweak the MotionControllerPawn blueprint to allow to navigation on the procedural mesh data visualization (it is rare in VR to navigate between different planes of varying height—which is of course necessitate in the navigation between areas in the bar chart).
- Create labels for each year and the "lowess mean" temperature that corresponds to it for each bar segment in the visualization, as well as box collisions atop each bar so that when the user is atop that bar, the label illuminates, allowing the user to better be able to tell which year they have climbed to.
- Create a 2D real time screen capture—effectivelly a mirror—in which the user can gauge how far they've climbed up the data.
- Use VCV Rack, a virtual modular synthesizer program, to create ambient music for the experience.