Example of long range LiDAR
[WIP]
15 |[WIP] 15 | https://www.insta360.com
16 |[WIP]
15 |Extra notes on using cameras outdoors? Link to my outdoor writeup?
16 |[WIP]
15 |Talk about the very basics of measuring a cameras field of view based on its lens. This can be good for estimating the amount of image you'll get at a given distance.
16 |Example of long range LiDAR
Example of what many LiDAR scanners look like
[WIP] 15 | Camera systems that may use multiple cameras or are cameras that are integrated with on-board computing power for computer vision.
16 | 18 |Need ideas here - 15 | Microphone Arrays as generalized cameras
16 |Notes about the actual physical limitations of cameras and technologies that will need to be further developed to overcome them?
17 |Freefly 15 | Full Spectrum Cameras (for UV and Infrared) 16 | "Bullet time" camera arrays (big freeze, A-1 array, other vendors, etc) 17 | David Rokeby's "Very nervous System" uses an array of light sensors to trigger effects
18 |[NEEDS IMAGE]
15 | 16 |LIDAR is a specialized imaging technique that has some similarities to the depth cameras discussed previously. LIDAR stands for Light Detection and Ranging and it has many applications in various industries, but it has caught on as a technology for interactive installations in the last decade.
17 |Suggested brands to investigate:
18 |These could have their own article and I have limited experience with them, but they are worth adding to the list for completeness. Motion capture systems are primarily used to capture the movements of performers that are then mapped to the skeletons of 3D models, essentiall puppeteering them. Motion capture systems can be camera-based or non-camera-based (like a suit covered in IMU's or inertial measurement units like the Xsens). We'll just cover the basic camera-based systems here.
15 | 16 |[WIP]
17 |The use cases for older camera technologies are much more rare and may only come up on certain productions that require the analog aesthetic in a way that can't be recreated with real time visual effects.
15 |It would be a large research project to dive into all of the older technologies, but we'll just stick with some noteable ones.
16 |Old consumer video cameras and security cameras are probably the most likely candidates for use. Their resolution is fairly low
17 |The biggest advantage that analog cameras used to have over their digital counterparts is that their latency from lens to screen was much lower because there weren't added layers of digital to analog conversion.
18 |
Render of an exploded view of a smartphone camera
[WIP]
15 | 16 |Another important thing to keep in mind is that while many artistically geared software environments do work off-the-shelf with various cameras and input types (typically with UVC, more exotic cameras aren't always plug and play. Make sure you do your research before making a big purchase. If the tool doesn't exist to pull in a video feed, there are a few technologies that can route video between applications or even over the network. The most common tools are Syphon for macOS and Spout for Windows, but there are other options out there as well.
17 |See also: CameraLink and CoaXPress
18 |[WIP]
15 |For the purposes of this writeup, we'll consider latency to be the delta or time between a person's action and the response or display from the software that captures their image.
16 |Most displays run at 60 frames per second, which means they are showing an image every (approximately) 16 milliseconds. When a camera is capturing an image, delay is introduced at several stages - the first and largest of importance is the camera's exposure time. Cameras need to leave their shutter open for a period of time to collect light on the sensor - the longer the shutter is open, the more light is collected. At 60fps, the ideal is a shutter speed of 1/60s, but shutter speeds can be longer to allow more light in for a darker environment. Additionally, latency can then be introduced by a camera's own buffering and processing, transmission to computer and subsequent writing to computer memory, and finally your own software's processing.
17 |Some cameras that advertise "low latency" are at around 35ms, which means in a best case scenario, you're seeing your reaction almost 2 frames after you've performed an action.
18 |Latency is especially noticable for audio applications and less so for visual. Musicians begin to have trouble with latency that is larger than 8-12ms. If you're planning on using a camera as a trigger for something like an interactive drumming station, you may find that the latency is unacceptable for certain kinds of playing.
19 |Here is a great Reddit thread on computer vision latency
20 |[NEED IMAGE]
15 |These cameras are typically employed in industries like manufacturing that require a high degree of stability and performance for performing computer vision tasks.
16 |One example would be the use of cameras and specialized software to monitor a fast moving automated assembly line and using the visual data to ensure every product looks correct. They can also be employed for things like processing produce - sorting ripe tomatoes from green tomatoes at very high speeds.
17 |This area of cameras is incredibly complex and we won't go into all of the specifics here, but you should definitely be aware that these specialized systems exist. The most common ones you will probably come across have already been mentioned, like machine vision infrared or thermal cameras. most of them range from having standard interfaces like USB and GigE, to proprietary cables and PCI cards for maximum data speed and low latency.
18 |Camera resources: 19 | Basler 20 | e-Con Cameras 21 | Allied Vision 22 | Point Grey/FLIR Blackfly 23 | OpenMV 24 | Elphel 25 | IDS 26 | Imperx 27 | Teledyne
28 |Wireless is a bit of a misnomer since almost any camera can be made wireless these days, and obviously even smartphone cameras count as a wireless camera. These would most commonly be used on film shoots for drones or remote monitoring by a director, but there may be limited usage for certain kinds of interactive installations and events. This technology is more about the interface that sends the image signal more than the actual image capture, so there isn't a ton to cover here.
15 |There are several devices out there for connecting any camera to a wireless interface that can stream the signal to a nearby computer over wifi, or over the internet via an integrated 4G/5G interface. The primary issues that will get introduced with wireless systems are: reliability, increased latency, and signal degredation as a side effect of video compression. Also - battery life is another beast to manage.
16 |Professional Wireless camera interfaces to investigate include:
17 | 22 |On the low end of cost for experimentation, smartphone cameras can be wirelessly streamed to desktop computers via various streaming apps. GoPro cameras are another classic consumer wireless camera that can be interfaced with. There are also a range of battery powered security cameras that may be the right choice depending on your application. 23 | For other resources, you can also look for devices marketed for wirelessly sending HDMI and RCA video signals instead of devices geared specifically towards making cameras wireless.
24 |Some wireless systems rely on existing wifi or other tradiitonal network infrastructure, and some generate and receive their own wireless signals. Many of them promise the ability to send signals over very long distances - like 750 to 1500ft (~230-460m) and up to 5000ft (about 1.5km). I would take those distance estimate with a grain of salt as they often require line of sight and very little interference, and reliable Wifi is always a weakpoint on many productions.
25 |Volumetric capture is another specialized approach that is very similar to motion capture systems, and the two are often used in tandem, particularly for modern cinematic video game animations. In its simplest form, volumetric capture is really a hybrid image capture approach that uses a regular camera matched with a depth camera, often in large 360º arrays of both, that allows the creation of 3D content. The more general term for volumetric capture is photogrammetry. There are also systems that just use large arrays of regular cameras instead of matching them with depth cameras. "Portrait" mode on modern smartphones is also a form of volumetric capture that combines two photos from two different lenses.
15 |Most of these professional systems require dozens of precisely cameras and an array of computers to ingest the massive amounts of data coming in. This data is then processed and turned into point clouds or meshes that can then be viewed in 3D rendering software or game engines. As of this writing, there are several systems and vendors out there that can achieve volumetric capture. Building a high quality full 360º system yourself would take considerable planning and budget.
16 |Depthkit is probably the most well known and accessible system for doing volumetric filmmaking. Depthkit, at its most basic, involves mounting a DSLR and a depth sensing camera together and doing some pre-alignment steps with their software system. As you capture both depth and color information, software is able to combine those two feeds into 3D content. Depthkit can be used by a wider group of people due to the lower barrier of entry on equipment and setup. The main trade offs are that the most common setup usually involves a single camera which means you typically get a limited amount of 3D information from the front surface of whatever you're filming.
17 |Larger "volcap" systems involve whole studios (although there are some systems that can be moved around). Microsoft's Mixed Reality Capture Studio is one example of the high end of these sorts of offerings. One version of their system uses 106 cameras to capture.
18 |Mention Volucams (https://twitter.com/BenSchwartzXR/status/1311704650475884545)
19 | 20 |Re-lighting, scene size limitations, complex scene limitations, movement limitations, bandwidth of final assets (mesh+color data per frame)
21 |[WIP]
22 |Advantage of monochrome cameras for accuracy over color images 15 | How to Measure the latency of your webcam with openCV 16 | DreamChip Cameras 17 | Netv2 video capture and overlay device 18 | DepthKit
19 |Huge thanks to Elliot Woods for a ton of thoughtful suggestions on how to expand the guide from its 2013 version. Kyle McDonald, Theo Watson, Zach Lieberman and Golan Levin for some great additional tips to help round out some of the suggestions here.
20 |Rough notes for for 2020 update: 21 | GOOGLE DOC FOR CAMERAS
22 |To-do for 2020 Update:
23 |
Various form factors of thermal cameras
http://en.wikipedia.org/wiki/Thermographic_camera
17 |These cameras are comparatively rare to see in use in interactive installations because they are still prohibitively expensive compared to most other ones. They aren't totally unobtainable ( roughly 1000-20000USD) but that higher price tag makes them a little less desirable for early exploration on projects. It's a shame because these cameras offer a lot of abilities that just aren't possible with the other kinds of cameras.
18 |There are various types of cameras to look at in this class, mostly pertaining to which part of the IR spectrum you're trying to see. You have the option of Long Wave IR, Mid Wave IR, and Short Wave IR. For thermal imaging, you'll mostly want to work with Long Wave IR, in the 7000-14000nm range.
19 |I have not personally used one of these cameras yet, but they have some properties that would be really amazing in the tool belt of people making interactive installations.
20 |Check out this guy doing some random demos with a thermal camera:
21 | 22 |Connection types: Most are made to be integrated into existing systems and either have proprietary connections or just output composite video. Some cameras communicate X/Y position of blobs.
23 |Resolution range: Some are very low resolution (the Thermitrack is 16 x 16px) and some are close to a VGA range, but don't expect to find HD thermal for cheap. You also get a variety of frame rates and contrast ranges.
24 |Thermal Camera Pros:
25 |Thermal Camera Cons:
35 |Further reading:
46 | 47 | 48 |
Example of a high speed camera from Phantom Imaging
[EDIT IN PROGRESS]
17 |Webcams can only go so far in terms of quality and other features, so in this section we'll cover other types of RGB cameras. Choosing something in this realm usually means you need something with higher image quality, or lower latency, or higher resolution, or some other custom need. To put a finer point on it, I'm bucketing these types of cameras into this section:
18 |Besides the increased price, the primary barrier to entry with these cameras is that they aren't as easy to plug in via USB and start working directly with the image, as with UVC webcams. Many cameras in this space either need a special driver or SDK that must be integrated into your software project. Cameras that don't have that option will typically output over some form of standard video cable like HDMI or SDI. HDMI and SDI can then be captured by specialized capture devices that can either be a specialized PCI capture card installed in a desktop, or they can connect over USB.
24 |One of the most common ones types for interactive installations is the DSLR, usually chosen for its improved image quality over a standard webcam. DSLR's have become much easier to integrate over the last few years, but they still have some challenges depending on the particular model.
25 |Capturing with a DSLR all day would be great because you'll get the best image for the price range, but those cameras are not always designed to be left on for extended durations. DSLR's can easily suffer from heating issues or auto-shutoff if left on for too long. A more robust production camera or a solid DV Cam is more ideal for long running installations that require a really good image (and can't use a webcam for whatever reason), but make sure to do a lengthy test run before installing. Note: if tested and implemented with care, some DSLR's can perform just fine for long installation runs (one person has mentioned one running for 6 months at 7 hours a day with no issues).
26 |DSLR's may need their SDK and some software massaging to work correctly in your environment, although some offer hit or miss HDMI output that would require a capture card. Canon's EDSDK for example has been ported to openFrameworks for easy access, and Magic Lantern Firmware.
27 |Other high end cameras also enjoy the benefits of working with cables that were designed with long cable runs in mind. Webcams over really long (10m/30ft+) USB cables can cause jitter or some other funky consistency issues, so again...test before installing whenever possible. Production quality cameras can work over HD-SDI for fairly long runs (around 200-500ft depending on who you ask), but for the really far stuff, you're best either going over a network if you don't mind latency, or going over fiber with a converter box. Fiber will get you considerably further, more than 2500ft if you need it. You'll also need a capture card for some of these, occasionally requiring a tower instead of a laptop. Adding length in almost any case is going to add to the potential for problems, so if it is something going over 200 or more feet, try to check your whole chain first.
28 |You're sometimes limited by the capture device and its support on your intended system. Some capture devices need special drivers or other magical mysticism to work within your intended environment, so be warned before going down this path...save your receipts.
29 |Further Reading:
30 |Frieder Weiss's writeup on using digital versus analog cameras and the latency issues involved - 2008)
31 |
Example of a DSLR camera
[NEEDS IMAGE]
15 |High-speed Cameras are a special class of cameras that can capture 250 frames per second or higher, even up to 250,000fps to several million fps in some cases. If the high-end machine vision cameras above are for real-time processing, I'm thinking of high-speed cameras more for offline recording and viewing - a slightly different workflow. High-speed cameras are fairly uncommon in interactive installations because the (current) limitations of physics mean you can't watch reality in real time and slow motion reality at the same time. Years ago, getting cameras that could capture higher than 60fps were fairly specialized and uncommon, especially for the consumer market. Now almost every flagship smartphone can record 120-240fps and sometimes even higher in burst modes. Some standard webcams can also get up to 120fps. The primary market for professional high speed cameras is for industrial purposes, like the high end machine vision cameras covered above, or for the film industry. Since these applications are fairly niche and low demand, these cameras tend to be incredibly expensive - ranging from around $500USD on the low end to $30,000USD to $50k+ on the high end.
16 |High-speed cameras typically work by continuously recording a circular buffer of frames into specialized on-board memory. When the camera receives a trigger to begin recording, that circular buffer is dumped and encoded into a regular video file that can then be downloaded off the camera or played back directly on the device. Depending on the resolution, frame rate and compression type, these files can be quite large, and can take time to download off the camera to process with software.
17 |Another noteable thing to know about high-speed cameras is their light requirements. Because of the incredibly fast shutter speed, High-speed cameras require a lot more light than a traditional video camera. A traditional camera may only need to capture a frame every 1/60th of a second, while a high speed camera needs to capture a frame every 1/1000th of a second, causing a drastic reduction in the amount of photons hitting the image sensor. Filming outside in direct sunlight is usually the best option, but if you need to capture indoors you will need to use the correct type of light. Incandescent lights, flourescent lights, and other older styles of lights tend to not work for high speed because they actually flicker at a rate faster than the naked eye can see (typically at the 60hz of a standard AC power source). Using an incandescent light source with a high speed camera will often reveal the light dimming and brightening instead of staying steady. For high speed cameras, very large lights that can't cool down quickly enough to flicker or LED light sources for film production tend to be preferred. As usual, do your research on the light since not everything is created equal. Here is another source on lighting for high-speed filming.
18 |Since high-speed cameras are really just great at pushing a lot of data through very quickly, youll find your tradeoff is usually between the desired resolution and your desired framerate. You can achieve very high framerates but at very low resolutions. These low resolution + high FPS videos can be useful for scientific work (like analyzing ballistics, for example) but not so much for providing a high quality clip for a user.
19 |Noteable high speed camera manufacturers are Phantom cameras from Vision Research, Photron cameras, and iX cameras, but their cameras can typically cost more than most interactive installation budgets can manage. Rental is often an option as well. The main issue you may run into with these cameras is actually interfacing with them. Because of their high cost and low usage in the interactive space, there often isnt a lot of prior knowledge out there about working with them and you need to have a camera before you can get documentation about their API's and such. Around 2014, Edgertronic entered the high-speed scene with their more affordable high-speed cameras. Edgertronic cameras are basically a specialized FPGA with a Linux computer for additional processing and control. I used several of these on an installation in 2014 for capturing footage of participants at 720p and 400fps and they performed fairly well and were easy to interface with via standard http requests and a browser interface. Several models have come out since then with various improvements. Most other cameras out there also interface via a network connection to some proprietary control software.
20 |
Example of what footage looks like when captured by an infrared camera
Example of depth map and infrared capture from an Azure Kinect
Infrared cameras have historically been one of the most important tools for artists working with interactive installations. In earlier years, an abundance of installations were set up in dark spaces for either aesthetic reasons or due to the technological limitations of display technology - aka dim projectors. In recent years, as installations move into well lit spaces and tracking algorithms improve, infrared camera relevance has started to shift. However, they still have some unique capabilities that come in handy for certain types of installations.
16 |As a quick primer, visible light goes from 400-700nm, but certain cameras have the ability to see infrared light from 700-900nm. This allows you to capture things that are not visible to the human eye, like IR tracking lights or a dark space illuminated by infrared light. Most cameras that aren't marketed as exclusively IR capable have infrared filters that block IR light, but in some cases, that filter can be manually removed. In order to limit a camera's ability to only see in infrared light, some may need filters that block all visible light - more info on that here
17 |There isn't a specific go-to camera for infrared these days. You can modify certain RGB cameras by removing their infrared filter..................[WIP]
18 |For the next level up, the other common one is the Point Grey Firefly. One tip for this one is to go ahead and buy only the monochrome version of this camera, as the color version has the additional color filter layers that can impact image quality and response. I stick with the firewire version of this camera for the bandwidth, but that can limit you on cable length (you can really only get firewire out about 10-15ft without using a repeater or amplifer, so keep that in mind when building equipment lists). The Firefly will also take a range of lenses that give you an insanely large aperture of around 1.4-1.2f and will be able to give you a much brighter image than a standard webcam. You WILL need a visible light filter for the Firefly to only get IR bands though. See Golan Levin's article on choosing the right IR filter for your application. In addition to the Point Grey, you can check out these cameras from Imaging Source.
19 |Connection Types: USB 2 (Playstation Eye or another webcam - modified), USB 3 (Point Grey Flea 3), Kinect's IR cam, Composite/Firewire (old Sonycam's with night vision)
20 |Resolution Range: You can generally get the same resolutions in IR that you can in regular webcams. There are plenty of tutorials out there for modifying standard webcams to only process IR light.
21 |**Range of effectiveness:**Generally the same as a webcam (1ft-50ft or .3-16m). Some IR cams like the Point Grey have a wider range of security camera lenses for various zoom ranges if you need to do tracking from far away.
22 |Infrared Pros:
23 |Infrared Cons:
31 |Optimal environment for round the clock reliability
38 |Room with a steady amount of IR light, either artificial or coming from a point source. Undesired sunlight can blow out an IR cam if the aperture isn't set right or if it doesn't have an automatic adjustment.
39 |Troublesome environments
40 |A dark room with insufficient IR light. A lot of normal light bulbs will still show up in IR even if they don't put out enough light to illuminate the IR spectrum, so keep an eye out for those in the background as pesky tracking distractions.
41 |Further Reading for IR cams
42 |Frieder Weiss's writeup on IR cameras and equipment -2008
43 |Other IR camera suggestions: 44 | Cheap Infrared CCTV cams work well, but you'll need a capture device 45 | Sony CCTV cam good in low light (needs light filter) 46 | The Sony M183 or M383 are classic low latency cams used for IR sensing, but are now discontinued, but check eBay.
47 |
Example of an RGB Webcam - the Logitech Brio