BGE / Creating 360 Video Panorama Viewer
Unknown
6:58 PM
360
,
bge
,
blender
,
blender game engine
,
image sequences
,
panorama
,
texture
,
viewer
I worked on 360 related project recently. It is involving stitching 360 panorama for 5 x GoPro cameras and I work under direction of Director Bruce Allan:
http://www.bruceallan.tv/
I decided to share a little bit of 360 panorama research notes that I wrote. I will talk a bit about Panorama creation and then moving into how to create 360 panorama viewer in BGE. The Blender Game Engine part is just some extra things I aspired to do, just because it is interesting to think how you would do it using Blender. Again, I have to mention that I am still a beginner with BGE, but I tried and this simple BGE setup seems to work.
Please give me feedback if you happened to have feedback inputs to improve this.
As of today, in this all digital era, we are actually presented with so many tools available already. Mostly affordable. You can so easily capture 360 still panorama using your smartphones. On the iPhone I use 360 app and Photosynth app, there are plenty of other apps that does this.
If you are more serious and using digital camera with fish eye lens, you want to look into something called Nodal Ninja. Or maybe Gigapan Epic Pro. Or something from GoPano or Kogeto Dot that utilizes some kind of Cone lens and you unwrap the video back into some kind of Cylinder projection:
http://www.gopano.com/
http://www.kogeto.com/say-hello-to-dot
Those technique that utilizes some kind of "warping" to take panorama in one shot usually gives less quality, but depends, maybe if the lens is really perfect and sharp (expensive) you can get really good result.
I guess it is still better to take multiple photos using a single camera or having multiple camera to take and record multiple shots and stitch them back together.
Anyhow, use the tools you got at hand to shot panorama.
Note: Normally to capture the whole 360 x 180 (full spherical panorama), we want to end up with Equirectangular or Spherical Projection Panorama that is laid flat. If you use iPhone 5 native panorama capture, for example, it takes something that is more like Cylindrical Panorama. It does not include the top (Zenith) and bottom (Nadir) of the panorama.
Taking it to another level, you might want to have a rig with multiple cameras, but it could be tricky getting things to work. And multiple cameras will not necessary give you better result, because the nodal pivot of camera will shift and will introduce parallax, no matter what.
4. ADD SOUND AND COMMENTARY
Adding background sound, music, etc, will add a lot to your panorama. Maybe use BGE 3D sound feature.
5. ADD STORY
Clickable panorama? Why not? Below is an inspiration:
http://labs.int3ractive.com/javascript/webgl/panorama/in2sg_office_panorama.html
6. MAKE A REAL 3D ENVIRONMENT
You could push it further by separating foreground and background object or even create a full 3D environment. This should be a good first exercise, just building a simple environment that loops day and night maybe.
7. MAKE A REAL GAME
So, this "360 panorama viewer" is certainly not a game. But you can think of some game ideas that uses 360 like this. There are already many apps for iPhone, iPad, and Android that utilizes the 360 gyro. BGE is great to create protoype games. I even feel like creating Android App now :]
Hopefully this post is useful.
WHAT I LEARN
Even for such a simple application, there are lot to consider:
- UI Design
- Some kind of Splash Screen / Help menu
- Additional feedbacks for user
- Thinking the best way for user to interact with the 3D scene
The best way to get better is definitely to learn about other people's works.
http://www.bruceallan.tv/
I decided to share a little bit of 360 panorama research notes that I wrote. I will talk a bit about Panorama creation and then moving into how to create 360 panorama viewer in BGE. The Blender Game Engine part is just some extra things I aspired to do, just because it is interesting to think how you would do it using Blender. Again, I have to mention that I am still a beginner with BGE, but I tried and this simple BGE setup seems to work.
Please give me feedback if you happened to have feedback inputs to improve this.
CAPTURING 360 PANORAMA
360 Panorama is an old art form that some photographers or cinematographers always like to experiment with. The idea is of course to capture the whole environment from a single point of view by stitching multiple photos. Simple enough, but it requires a lot of preparation to get that perfect panorama.As of today, in this all digital era, we are actually presented with so many tools available already. Mostly affordable. You can so easily capture 360 still panorama using your smartphones. On the iPhone I use 360 app and Photosynth app, there are plenty of other apps that does this.
If you are more serious and using digital camera with fish eye lens, you want to look into something called Nodal Ninja. Or maybe Gigapan Epic Pro. Or something from GoPano or Kogeto Dot that utilizes some kind of Cone lens and you unwrap the video back into some kind of Cylinder projection:
http://www.gopano.com/
http://www.kogeto.com/say-hello-to-dot
Those technique that utilizes some kind of "warping" to take panorama in one shot usually gives less quality, but depends, maybe if the lens is really perfect and sharp (expensive) you can get really good result.
I guess it is still better to take multiple photos using a single camera or having multiple camera to take and record multiple shots and stitch them back together.
Anyhow, use the tools you got at hand to shot panorama.
Note: Normally to capture the whole 360 x 180 (full spherical panorama), we want to end up with Equirectangular or Spherical Projection Panorama that is laid flat. If you use iPhone 5 native panorama capture, for example, it takes something that is more like Cylindrical Panorama. It does not include the top (Zenith) and bottom (Nadir) of the panorama.
Taking it to another level, you might want to have a rig with multiple cameras, but it could be tricky getting things to work. And multiple cameras will not necessary give you better result, because the nodal pivot of camera will shift and will introduce parallax, no matter what.
STITCHING PANORAMA
Panorama Tools of my choice:- PTGui - paid license --> can do batch really easily
- Hugin - open source --> actually very similar to PTGui, can do batch but little bit more tricky.
- Microsoft ICE - kind of open source --> see also Photosynth
Useful and inspiring website links related to Panorama:
- http://www.hdrlabs.com/tutorials/index.html
- http://en.wikipedia.org/wiki/Equirectangular_projection
- http://punkoryan.com/2011/02/08/shooting-360-degree-video-with-four-gopro-hd-hero-cameras
- http://howtoshoot360.com/
- http://janvrsinsky.com
- http://www.mediavr.com/blog/
- http://tourwrist.com/
- http://photosynth.net/
- http://www.panoramio.com
REFINING PANORAMA USING BLENDER COMPOSITING
The resulting panorama stitch done using panorama stitching tool may or may not be perfect. Actually sometimes you need to compromise because getting perfect 360 is tricky, especially if you are capturing using multiple cameras at the same time, where the viewpoint or nodal of the camera is shifted which causes Parallax Error, especially noticable around the FOREGROUND objects. For BACKGROUND, or faraway usually you should not have issue.
Sometimes you also have some object moving, that needs to be cleaned up either using something like Photoshop or perhaps using Compositing package, which is useful if your panorama is actually a set of image sequences.
PANORAMA VIEWER USING BGE (BLENDER GAME ENGINE)
Now, this is the interesting bit that I like to share.
It is rather simple setup, all 3D artist have done setting up for 3D environment sphere.
But using BGE, you can have a 360 panorama viewer (still) or 360 video sequence panorama viewer (slightly more advanced) and interaction. Interaction is the big word here, you can give control and simple mouse interaction for users that know nothing about 3D and Blender.
Super easy, super simple, why not create an executable program? Below is all the steps I noted:
It is rather simple setup, all 3D artist have done setting up for 3D environment sphere.
But using BGE, you can have a 360 panorama viewer (still) or 360 video sequence panorama viewer (slightly more advanced) and interaction. Interaction is the big word here, you can give control and simple mouse interaction for users that know nothing about 3D and Blender.
Super easy, super simple, why not create an executable program? Below is all the steps I noted:
1. THE 3D SCENE
All we need is a simple polygon Sphere mesh with properly laid out UV as the 360 panorama background and then a Camera that we control using BGE. Really simple right?
 |
| Add some 3D objects to add extra layer of interest. |
For a single Still 360 Panorama, all we need is to map the Equirectangular image into the Sphere. And it should do the job.
You can further create a control inside BGE that "switch" the Panorama environment sphere as you please. Perhaps you want to give the user more than just 1 panorama. You can have some 360 panorama gallery.
2. CONTROL FOR VIEWING (MOUSE LOOK SCRIPT)
For this purpose, we only need a single camera that is locked in 0,0,0 position, world center axis. We don't need to get too complicated.
What kind of control we give for this camera? The most common is to give Mouse Look control, where user can look around by just moving the mouse.
Mouse movement should control "head pivot" and then extra control AWSD can be added to give smooth linear rotation to the left or to the right. And maybe add ZOOM control as well that is linked to animation action of Camera Focal Length.
For this setup in BGE, I learn by looking at this website for tutorial on First Person Mouse Look:
The script provided above is for BGE older version, luckily there is an updated version that works:
With the updated script, the camera First Person Mouse Look should work as expected. I have tried it and I confirm the script is working. Except for one thing: "cap" does not seem to work, which means the Mouse Look tend to go over 90 degree and it spins around without limit.
Alternative to the script above is using this alternative script called "Ultimate Mouse Look":
You simply apply it as SENSOR: Always (pulse on) and plug into CONTROLLER: Python. It will work.
For our purpose I switch off the MOVE so that the camera is locked. All user need for 360 panorama viewing is basically like "head pivot movement and left-right rotation like an owl".
NOTE: I got a feedback that this setup will take over mouse control while running. Now this is something I didn't think at first, but probably need to find a way around it.
NOTE: I got a feedback that this setup will take over mouse control while running. Now this is something I didn't think at first, but probably need to find a way around it.
3. TURNING IT INTO VIDEO PANORAMA SEQUENCE
Now, this is probably the most tricky bits but also exciting part because we are adding a feature that allow for 360 video panorama (panorama sequence) for full immersive experience.
For my project, I actually need a setup that can playback 360 video panorama. Luckily, this feature is actually provided by BGE!
Recent version of BGE can playback video in realtime. It can also playback webcam stream, I heard, but I have not tested it yet. But for video, it is really possible.
http://www.blender.org/documentation/blender_python_api_2_59_0/bge.texture.html
All you need for this is to look into Game Engine Visual, Video Texture example that you can get from here:
http://www.blender.org/development/release-logs/blender-261/blender-261-demo-files/
In that examples, there is one example blend file that show a 3D game scene that plays VIDEO in REAL-TIME. The movie is a trailer in OGG video format:
In that examples, there is one example blend file that show a 3D game scene that plays VIDEO in REAL-TIME. The movie is a trailer in OGG video format:
Come to pleasant surprise the "video texture" actually warp-able. You can simply map it into Sphere and this is really just perfect for our purpose in creating 360 panorama video viewer.
You simply use that example asset in your own panorama scene and thing should work perfectly!
NOTE: For this 360 project I am doing, I firstly created a video panorama sequence MOV, then convert it into OGG format that BGE accepted.
You simply use that example asset in your own panorama scene and thing should work perfectly!
NOTE: For this 360 project I am doing, I firstly created a video panorama sequence MOV, then convert it into OGG format that BGE accepted.
4. ADD SOUND AND COMMENTARY
Adding background sound, music, etc, will add a lot to your panorama. Maybe use BGE 3D sound feature.
5. ADD STORY
Clickable panorama? Why not? Below is an inspiration:
http://labs.int3ractive.com/javascript/webgl/panorama/in2sg_office_panorama.html
6. MAKE A REAL 3D ENVIRONMENT
You could push it further by separating foreground and background object or even create a full 3D environment. This should be a good first exercise, just building a simple environment that loops day and night maybe.
7. MAKE A REAL GAME
So, this "360 panorama viewer" is certainly not a game. But you can think of some game ideas that uses 360 like this. There are already many apps for iPhone, iPad, and Android that utilizes the 360 gyro. BGE is great to create protoype games. I even feel like creating Android App now :]
EXPORT AND PUBLISH
Next thing you need to do is just to activate Blender Add-On to export out your "game" or " 360 panorama viewer" as stand alone executable. I tested on PC and Mac. On PC, it becomes EXE with some files. I recommend you to save the executable inside a folder, so that all the files are placed in that folder. On Mac, it becomes an App bundle.
One thing, it would be nice if I could create feature that allow INPUT from users, allowing them to supply their own 3D panorama. I have to check whether this is possible or not. For now, I just pack all the assets into BLEND file before exporting as executable. File - External Data - Pack into .blend files..
One thing, it would be nice if I could create feature that allow INPUT from users, allowing them to supply their own 3D panorama. I have to check whether this is possible or not. For now, I just pack all the assets into BLEND file before exporting as executable. File - External Data - Pack into .blend files..
LITTLE ISSUE
One issue I found is with BGE stand-alone executable keep crashing whenever I use texture. I am still wondering if this is happening on my local computer only.
Anyhow, the on-going thread is here:
http://blenderartists.org/forum/showthread.php?269315-Game-Engine-Movie-Image-Sequence-UV-Mapping
UPDATED 2012.10.18:
The executable is working properly with latest Blender 2.64a build.
UPDATED 2012.10.18:
The executable is working properly with latest Blender 2.64a build.
Without that crashing, I could already share a BGE stand alone showing the finished 360 viewer.
CONTROLS:
A = rotate/turn left
D = rotate/turn right
W = toggle zoom in and out (focal length)
Mouse = rotate around pivot
- This is PC version EXE: (28 MB)
https://dl.dropbox.com/u/3654459/BLENDERSUSHI/360_exe.zip
- This is MAC version APP: (21 MB)
https://dl.dropbox.com/u/3654459/BLENDERSUSHI/360.app.zip
CONTROLS:
A = rotate/turn left
D = rotate/turn right
W = toggle zoom in and out (focal length)
Mouse = rotate around pivot
LOOKING FORWARD
I personally would like to turn this into an app for iOS or Android. 360 video panorama viewer that utilizes the gyro accelerator, really simple to do I think and still quite fun. At the moment, I do not have the knowledge. Developers reading this post will probably be inspired to do it. It is a rather simple app, but I think app like that is really fun and useful.
Hopefully this post is useful.
WHAT I LEARN
Even for such a simple application, there are lot to consider:
- UI Design
- Some kind of Splash Screen / Help menu
- Additional feedbacks for user
- Thinking the best way for user to interact with the 3D scene
The best way to get better is definitely to learn about other people's works.
