GetUserMedia: Using the Media Capture and Streams API

Final product image — What You'll Be Creating

Today I’d like to experiment with the Media Capture and Streams API, developed jointly at the W3C by the Web Real-Time Communications Working Group and the Device APIs Working Group. Some developers may know it simply as getUserMedia, which is the main interface that allows webpages to access media capture devices such as webcams and microphones.

You can find the source code for this project on my GitHub. Additionally, here’s a working demo for you to experiment with. In the latest Windows 10 preview release, Microsoft added support for media capture APIs in the Microsoft Edge browser for the first time. Much of this code was taken from the Photo Capture sample that the Edge dev team produced at their test drive site.

For those of you who want to dive a bit deeper, Eric Bidelman has a great article at HTML5 Rocks which goes into the storied history of this API.

Getting Up to Speed

The getUserMedia() method is a good starting point to understand the Media Capture APIs. The getUserMedia() call takes MediaStreamConstraints as an input argument, which defines the preferences and/or requirements for capture devices and captured media streams, such as camera facingMode, microphone volume, and video resolution.

Through MediaStreamConstraints, you can also pick the specific captured device using its deviceId, which can be derived from the enumerateDevices() method. Once the user grants permission, the getUserMedia() call will return a promise with a MediaStream object if the specific MediaStreamConstraints can be met.

All of this without needing to download a plugin! In this sample we’ll be diving into the API and creating some neat filters on the video and images we’ll capture. Does your browser support it? Well getUserMedia() has been around since Chrome 21, Opera 18, and Firefox 17, and is now working in Edge.

Feature Detection

Feature detection involves a simple check for the existence of navigator.getUserMedia. It’s a big project to check in each browser, so I’d suggest simply using Modernizr to check for it. Here’s how it works:

if (Modernizr.getusermedia) {
  var getUM = Modernizr.prefixed('getUserMedia', navigator);
  getUM({video: true}, function( //...
  //...
}

Without Modernizr, as demonstrated in this sample, you’d have to use:

navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia;
 
if (!navigator.getuserMedia) {
  Console.log('You are using a browser that does not support the Media Capture API');
}

The Video Player

In our HTML you can spot the video tag towards the top of the page. You’ll notice that it’s also set to autoplay. Without that, the video would permanently be frozen on the first frame.

<div class="view--video">
    <video id="videoTag" src="" autoplay muted class="view--video__video"></video>
</div>

Gaining Access to an Input Device

There currently isn’t a source for the media set, but we’re going to inject that source via JavaScript shortly.

This new functionality can enable a number of new opportunities for developers, but it can also be a security risk for the end user. Therefore, the first thing you’ll notice when you launch this web app is that it requests permission to use your webcam.

GetUserMedia accepts a few parameters. The first is an object specifying the details and requirements for each type of media you want to access. For access to the webcam, the first parameter should be {video: true}. Additionally, to use both the microphone and camera, pass {video: true, audio: true}.

Supporting Multiple Webcams

This is where things really get interesting. We are also using the MediaDevices.enumerateDevices method in this sample. This collects information about the media input/output devices available on your system, such as microphones, cameras, and speakers. This is a promise which will return several properties, including the kind (type) of device, such as videoinput, audioinput, or audiooutput. Additionally, it can generate a unique ID in the form of a string with a unique ID (videoinput: id = csO9c0YpAf274OuCPUA53CNE0YHlIr2yXCi+SqfBZZ8=), and finally a label to describe the device, such as FaceTime HD Camera (Built-in). This is still an experimental technology though, and isn’t even listed on CanIUse.com yet.

Setting the Source on the Video Player

In the initalizeVideoStream function, you can see that we are getting the video tag from our page and setting its source to the stream we are passing in. The Stream itself is a blob. If the browser doesn’t support the srcObject attribute, it falls back to create a URL out of the media stream and sets that.

    // initializeVideoStream() - Callback function when getUserMedia() returns successfully with a mediaStream object
    // 1. Set the mediaStream on the video tag
    // 2. Use 'srcObject' attribute to determine whether to use the standard-based API or the legacy version
 
    var initializeVideoStream = function(stream) {
        mediaStream = stream;
 
        var video = document.getElementById('videoTag');
        if (typeof (video.srcObject) !== 'undefined') {
            video.srcObject = mediaStream;
        }
        else {
            video.src = URL.createObjectURL(mediaStream);
        }
        if (webcamList.length > 1) {
            document.getElementById('switch').disabled = false;
        }
    };

Applying CSS Filters

I’m not very good at taking photos, so I often rely on the filters that Instagram provides for me. But what if you could apply your own filters to your video or static image? Well you can!

I’ve created a simple function for the video feed, which allows me to apply CSS filters in real time. The one for the image is nearly identical.

    // changeCssFiltersOnVid() - Cycle through CSS filters applied to the video stream
    // 1. Grab a reference to the video tag
    // 2. Keep the original CSS classes while still adding the filters
    // 3. Loop through all of the filters
 
    var changeCssFilterOnVid = function () {
        var el       = document.getElementById('videoTag');
        el.className = 'view--video__video';
 
        var effect = filters[index++ % filters.length]
        if (effect) {
            el.classList.add(effect);
            console.log(el.classList);
        }
    }

At the top of the class I have an array of filters to loop through. They are stored as a string, which corresponds to classes with identical names in CSS.

// CSS filters var index = 0; var filters = ['grayscale', 'sepia', 'blur', 'invert', 'brightness', 'contrast', ''];

And in the CSS:

/* image * video filters */
.grayscale {
    -webkit-filter: grayscale(1);
    -moz-filter: grayscale(1);
    -ms-filter: grayscale(1);
    filter: grayscale(1);
}
 
.sepia {
    -webkit-filter: sepia(1);
    -moz-filter: sepia(1);
    -ms-filter: sepia(1);
    filter: sepia(1);
}
 
.blur {
    -webkit-filter: blur(3px);
    -moz-filter: blur(3px);
    -ms-filter: blur(3px);
    filter: blur(3px);
}

You can see more examples of how this works, and change values in real time, at the Edge test drive page.

Saving Images

Digging through the code, you may see some other features that you are not immediately familiar with. The first thing that grabbed my eye was navigator.msSaveBlob. The Blob constructor allows you to easily create and manipulate a blob (basically a file) directly on the client. It is supported in IE 10+.

The msSaveBlob method allows you to save this blob object (in this case, our snapshot image) to disk. It also has a sibling method, msSaveOrOpenBlob, which also allows you to open the image from within the browser.

 // savePhoto() - Function invoked when user clicks on the canvas element
 // 1. If msSaveBlob is supported, get the photo blob from the canvas and save the image file
 // 2. Otherwise, set up the download attribute of the anchor element and download the image file
 
    var savePhoto = function() {
        if (photoReady) {
            var canvas = document.getElementById('canvasTag');
            if (navigator.msSaveBlob) {
                var imgData = canvas.msToBlob('image/jpeg');
                navigator.msSaveBlob(imgData, 'myPhoto.jpg');
            }
            else {
                var imgData = canvas.toDataURL('image/jpeg');
                var link    = document.getElementById('saveImg');
                link.href   = imgData;
                link.download = 'myPhoto.jpg';
                link.click();
            }
            canvas.removeEventListener('click', savePhoto);
            document.getElementById('photoViewText').innerHTML = '';
            photoReady = false;
        }
    };

If the browser supports the method, it will cut down the amount of code we need to write to save an image.

Where Else Can We Go From Here?

This is only the beginning. We can also utilize WebGL with this, which allows for even more filters to be applied, as well as having a real-time video/audio feed built into highly interactive environments. Maybe that will be my next project…

Additionally, you can tie into the Web Audio API to apply frequency modulation to your audio output. This sample from the Web Audio tuner illustrates it well. Some folks are more about visual learning, so check out this Microsoft sample, too.

Finally, as mobile browsers continue to adopt more of this technology, you’ll be able to use these JavaScript APIs to tie into the underlying hardware and have this working on mobile devices, regardless of the platform. It’s a great time to be a web developer, and hopefully after you’ve used this, you’ll understand why I’m so excited to be a part of it.

Feel free to check out the source code, and to see a working sample.

More Hands-On With JavaScript

Microsoft has a bunch of free learning on many open source JavaScript topics, and we’re on a mission to create a lot more with Microsoft Edge. Here are some to check out:

Microsoft Edge Web Summit 2015 (a complete series of what to expect with the new browser, new web platform features, and guest speakers from the community)
Best of //BUILD/ and Windows 10 (including the new JavaScript engine for sites and apps)
Advancing JavaScript Without Breaking the Web (Christian Heilmann’s recent keynote)
Hosted Web Apps and Web Platform Innovations (a deep-dive on topics like manifold.JS)
Practical Performance Tips to Make Your HTML/JavaScript Faster (a seven-part series from responsive design to casual games to performance optimization)
The Modern Web Platform Jump Start (the fundamentals of HTML, CSS, and JavaScript)

And some free tools to get started: Visual Studio Code, Azure Trial, and cross-browser testing tools – all available for Mac, Linux, or Windows.

This article is part of the web dev tech series from Microsoft. We’re excited to share Microsoft Edge and the new EdgeHTML rendering engine with you. Get free virtual machines or test remotely on your Mac, iOS, Android, or Windows device @ http://dev.modern.ie/.

HIGHLIGHTS OF THE DAY