There is this invisible field of energy around us all the time, in fact, all living organisms generate this field of energy constantly. Fundamentally it is the stretch of electromagnetic waves within and beyond living bodies. In scientific terms, we call it electromagnetic field or EMF. The human EMF can be detected several feet away from the human body and whenever we come close to other humans or living organisms we interfere with their EMFs. In physics, this phenomenon is called interference or Electromagnetic Interference (EMI). In the natural world, the EMIs have been present throughout the evolution of life on earth and existed in harmony while benefiting the natural world. However, the man-made interference caused by man-made devices is creating an electromagnetic disturbance and this disturbance is altering the natural world. Several reports are explaining how EMF of telecommunication and electric cable deployment in coastal waters are negatively altering the behaviour of marine life.
The electromagnetic field (EMF) which is generated by electromagnetic waves is an invisible phenomenon. The human eye can only see a small portion of this electromagnetic spectrum. The rest of it is invisible. However, computer vision and other scientific apparatus can help us understand and visualize this invisible world. In my current project, I am aiming to visualize the invisible electromagnetic disturbance. The human gestures in this interactive installation are the symbolic representation of man-made electromagnetic disturbance and the deformation of biomorphic Voronoi patterns represents non-human living organisms.
The piece is trying to produce a visual representation of the invisible electromagnetic disturbance through the encounter of gestures and Voronoi patterns. Using capturing systems and 3D imaging from video game technology combined with a dance performance the installation allows us to offer an immersive experience to the audience. The hand gesture coherently interferes with 3D image, creates a strong relationship between human and the other world.
Capturing Human gesture
The technical challenges of interaction in the video game industry are similar to interactive art installation. I am using Microsoft Kinect (Depth Camera) Version 1. The Kinect V1 SDK allows us to track the user’s joint movement. However, Kinect V1 does not have any official Unity3D plugin and therefore there is no official documentation. Although, I was able to find third party working plugin which allowed me to successfully capture the user’s joint movement.
The code
Kinect joint tracking in unity3D outputs the values in pixels that were easy to use. As shown in the code I am using Camera.ScreenPointToRay function to cast a ray from the camera to the target surface. Everything worked fine but since it was a mirrored image the y-axis coordinates were reversed. that is why I have used the following approach:
Camera.main.ScreenPointToRay(new Vector2 (posColor.x- 10f, KinectWrapper.Constants.ColorImageHeight - posColor.y - 10f));
the -10f is an arbitrary value of the red overlay object in a pixel which I am using to make it roughly centred to the point where the force is being applied.
The code logic in “MeshDeformerAgent.cs” is derived from two different scripts, Kinect SDK Unity3D plugin and a Mesh deforming script. The mesh deforming script was originally using mouse input to cast the Ray but I have changed it to Kinect joint coordinates.
The Mesh
Inspired by an interactive Hakanaï performance I was looking to use a pattern of a rectangular grid. During my exploration and experimentation, I was using a JPEG image. The file was applied to the Albedo map with a tiling size corresponding to screen measurements. This approach was ok but at the point of deformation, the albedo map was pixelating. The real struggle was to find something light yet beautiful to interact with. After experimenting with many 3D mesh generation tools, I ended up with Rhino3D. Grasshopper with Rhino3D provides a sophisticated way to generate beautiful patterns in the mesh. Sophisticated but processor-heavy and fatiguing. I spent multiple sessions to find the best tutorial. The available tutorials were not quite fulfilling my requirement. They were exporting a large OBJ file almost 150MB and I was looking for something in KBs.
There are several things which didn’t work to my expectation. This piece was intended for media projection. In an ideal situation, I was looking for at least two media projectors mapped towards two adjacent walls. Unfortunately, it was not possible during the pandemic and I had to find a different way to present. The alternative path which I choose to present the piece didn’t work either. In my current approach, I am using a blue screen to remove the background from the dance performance video. The dance video is filmed separately and the unity GameView screen is captured separately and then combined in Adobe Premier. However, the final video does not look impressive.
The distortion/deformation effect of the Voronoi patterns is a little off. I think it is because of the void spaces in Voronoi patterns. When the Ray hits the Voronoi mesh it passes through from the holes in the pattern, causing an inconsistent deformation effect.
The joint coordinates work fine when the unity GameScreen is windowed but when I maximize the screen the player joint coordinates do not respond to screen size. I think this is something to do with Camera resolution (ColorImageHeight, ColorImageWidth).