GetUserMedia Constraints explained

By Olivier Anguenot

Published in api

April 24, 2023

8 min read

GetUserMedia

What are constraints?

Constraints Grammar

Why constraints?

Playing with constraints

Constraints error

Interesting new stuff

Conclusion

The getUserMedia API was the first API released when WebRTC landed in the browser: Here is the article from Chrome and the one from Firefox.

This API allows a website to get audio and video media from a device connected to the computer and let your application manipulate it.

But even if this API seems easy to use, there is a lot behind it that I tried to understand to use it in the best way.

GetUserMedia

What a strange name for an API that retrieves media from a device…

getUserMedia is not a getter API that returns media instantly. It is a Promise that resolves (asynchronously) only if:

(1) At the user level: The user accepts or has accepted the permission requested by the browser to allow the website to access the audio and/or video device(s).
(2) At the application level: Constraints requested by the application when requesting media can be satisfied. This includes to overcome any issues that may arise when using physical (or virtual) devices.

Depending on the browser and the previous choice, this API may or may not trigger a pop-up window that forces the user to accept or decline the authorization. This first step can be optional if your application (website) has already been authorized for this device (e.g. in Firefox) or for all devices (e.g. Chrome).

Note: This article focuses on part (2) which is the use of constraints by taking authorization for granted.

Note: Personally, I’m not a big fan of this mix of concerns. Since we now have a Permissions API, I would have preferred that authorization be handled by a permission API as well so that getUserMedia only focuses on accessing the media without having to display the authorization popup (as with the native).

What are constraints?

When calling the getUserMedia API, constraints are the given parameters (a JSON object) that force the media to follow a desired behavior.

These constraints affect:

(a) The type(s) of the media to use: (Mandatory constraint). The possibility to have or not to have a media of a specific type (or type). For example, the application can ask to have only video media, not audio media.
(b) The origin of the media:(Optional). This is the ability to target as input any device identified by the browser, not just the default system device,
(c) The settings of the media: (Optional). Finally, it is the possibility to configure the settings managed by the selected device. For example: the size of the video.

The use of constraints is a way for the application to match the media to its need. For example, it is not necessary for the application to ask for audio media if the application only wants to take a picture.

Then, and depending on the execution of the request, the result of the getUserMedia API call may be:

a media with audio and video. Concretely, the result is a MediaStream containing 2 MediaStreamTrack (one for audio and one for video)
A media with audio or video : Concretely, the result is a MediaStream containing 1 MediaStreamTrack (for audio or for video)
No Media: When the promise cannot be fulfilled (error).

Constraints Grammar

When using constraints, there is a little grammar to understand, based on the following rules: exact, ideal, min, max and a separate section called advanced. Writing a constraint is like writing an SQL query (I hope not…).

exact is a mandatory rule to satisfy. If the constraint cannot be satisfied, the promise fails, which means you will not be able to access the media or apply a new constraint to it.
min and `max are mandatory range rules to satisfy. If the device is not able to provide a value within the specified range (e.g. a video with a width greater than 1024px), the promise fails and this leads to the same result as the previous case.
ideal is an optional-to-satisfy rule. The device will try to satisfy the request but if it cannot and returns a different value for the constraint, the promise will not fail. This is equivalent to writing a key:value directly without the word ideal.
advanced is an internal section for mandatory-with-no-failure rules :-). It’s complicated to explain…, but all rules placed here (treated as exact) are executed after all other exact rules and before all ideal rules. The difference here is that unfulfilled constraints are rejected without fail. This is a way to test different values and get the best one…

Note: this grammar targets the getUserMedia API as well as the applyConstraints API.

Some hints:

A call to getUserMedia can contain multiple constraints targeting different media types (audio, video),
Each constraint can have several mixed rules. E.g. You can ask for a minimal width of 800 and an ideal of 1024: {video: {width: {min: 800, ideal: 1024}}}.
When using a rule mix in a constraint, the mandatory rules to be satisfied are processed first, then the optional rules to be satisfied if necessary. Using exact with an optional rule (min, max or ideal) does not make sense: Either the exact rule is satisfied, or the promise fails.
When using a mixture of rules in a constraint, if one of the mandatory rules cannot be satisfied, the promise fails,
When using multiple constraints, if one of them has a mandatory satisfaction rule that cannot be satisfied, the promise fails,
When using optional-to-satisfy rules only in one or more constraints, the resulting media may not respect the requested constraints.
The advanced section is an array of constraints (to be able to put the same constraint several times),
In the advanced section, the constraints never miss the promise,
In the advanced section, the order is important. You can have several times the same constraints with different rules. The first match wins.

Why constraints?

Because getUserMedia requires at least one minimum viable constraint containing the type(s) of a media to capture.

You cannot call getUserMedia without any parameters, with an empty JSON or with a JSON that does not have at least one audio or video property with a true value. This fails. This is to avoid any extra and misleading media. By default, there is no audio or video.

So you must add at least one media with a value of true.

// Minimal constraint: select the kind of media you want to use.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

When using a type(s) constraint only, origin and behavior are the default properties:

For origin, the user’s primary device or the system default device is used. The exception goes to Firefox where you have to choose which devices to use (authorization is tied to the selected devices).
For settings, each binding property uses a default value given by the system. There is some consensus around the video size (640x480 by default for desktops), the video refresh rate (30fps), but depending on the browser and hardware (desktop, mobile), the defaults may be different.

Constraints represent a “behavior” requested by the application to provide the best possible experience.

Playing with constraints

Browser constraints versus device constraints

Just because the browser offers support for constraints does not mean that these constraints are supported by the underlying device used and vice versa.

There are 2 levels: Constraints supported at browser level and constraints supported at device level.

Because the browser must pass a constraint to devices, it must understand and support it. To find out which constraints are supported by the browser, you can use the getSupportedConstraints API:

const supportedBrowserConstraints =
  navigator.mediaDevices.getSupportedConstraints();

// supportedBrowserConstraints = {deviceId: true, width: true, height: true, zoom: true, ...}

Unfortunately, when comparing Chrome and Firefox, you will see a large gap between the two browsers, where few constraints are supported by Firefox.

Note: The getSupportedConstraints API returns all constraints supported by browsers for the getUserMedia API and for the getDisplayMedia API. Some of them can only be used in the right context.

For the device, since there is no actual device management, except for the MediaDeviceInfo JSON structure, the application needs to query the obtained MediaStreamTrack for supported constraints using the getCapabilities API.

const supportedDeviceConstraints = track.getCapabilities();

// supportedDeviceConstraints = {width: {min: 1, max: 1280}, height: {min: 1, max: 720}, ...}

getCapabilities lists the constraints supported by the device. This is a bit strange because you have to get the track to know … what is supported by the device.

The API returns not only the list of supported constraints, but also the accepted values or ranges. For example, for dimension, you can infer whether the camera supports a 4K stream or not.

Current constraints values

Knowing the supported constraints is not enough, you need to know the current value of each of them. This can be done using the getSettings() function.

const currentSettings = track.getSettings();

// currentSettings = {width: 1280, height: 720, frameRate: 30, ...}

Note: Keep in mind that the constraints, if met, affect the local stream. What is received by the recipient(s) may be different depending on the media pipeline used, network conditions or the media server in the middle.

Constraints used

There is one last API that returns the constraints requested/accepted by the device: getConstraints. Using this function, you know how your application currently influences the “default” values.

const currentConstraints = track.getConstraints();

// currentConstraints = {deviceId: 0e05387a88dec20949ff8d8d18ee288ed4d7271a8d4380b497feb1432300c4bd, ...}

Applying new constraints

At any time, you can apply new constraints to the live track using the applyConstraints function. This can be a handy method of adapting the media to a change, for example if the system is suffering from CPU, your application may ask to get a stream with a lower resolution.

try {
  await track.applyConstraints({
    width: { exact: 640 },
    height: { exact: 480 },
  });
} catch (e) {
  // The constraints can't be applied to the device
}

Note: Be careful when using this method, all previously defined and not added constraints are lost and the values go back to default. It would therefore be wise to call getConstraints() and use the result as input to applyConstraints after modifying it.

Note: You may notice that you don’t need to use the audio or video properties because the track is already typed. So there is no error possible.

Common constraints

For selecting a specific device, you need to use the deviceId constraint. In the following code, I try to get a specific microphone

const constraints = {
  audio: {
    deviceId: {
      exact: "0e05387a88dec20949ff8d8d18ee288ed4d7271a8d4380b497feb1432300c4bd",
    },
  },
};

const stream = await navigator.mediaDevices.getUserMedia(constraints);

// Stream contains an audio track captured from my microphone.

Knowing that my microphone can capture in stereo, I can update my query with the following

const constraints = {
  audio: {
    deviceId: {
      exact: "0e05387a88dec20949ff8d8d18ee288ed4d7271a8d4380b497feb1432300c4bd",
    },
    channelCount: 2,
  },
};

const stream = await navigator.mediaDevices.getUserMedia(constraints);

// Here, the audio track will be in stereo.

For the video part, you can add width and height constraints as well as a framerate.

const constraints = {
  video: {
    deviceId: {
      exact: "0e05387a88dec20949ff8d8d18ee288ed4d7271a8d4380b497feb1432300c4bd",
    },
    width: { min: 800, ideal: 1280 },
    height: { min: 600, ideal: 720 },
    frameRate: 30,
  },
};

const stream = await navigator.mediaDevices.getUserMedia(constraints);

// Here, the device tries to give a HD video at 30fps but if it can't, it should return a resolution at least of 800x600.

Or if you want to stick to a specific aspect ratio, you can use the aspectRatio constraint like in the following:

const constraints = {
  video: {
    deviceId: {
      exact: "0e05387a88dec20949ff8d8d18ee288ed4d7271a8d4380b497feb1432300c4bd",
    },
    width: { min: 800, ideal: 1920 },
    aspectRatio: { exact: 4 / 3 },
  },
};

const stream = await navigator.mediaDevices.getUserMedia(constraints);

// Here, the device tries to give a video at 1920x1440 (due to the aspectRatio) which is not managed by my camera, but I got a video track with a resolution of 1440x1080.
// The same request using the exact rule instead of the ideal one for the video makes the promise fail.

Using the advanced section, we can try to have the best possible frameRate without failing the promise:

const constraints = {
  video: {
    deviceId: {
      exact: "0e05387a88dec20949ff8d8d18ee288ed4d7271a8d4380b497feb1432300c4bd",
    },
    width: { exact: 1920 },
    height: { exact: 1080 },
    advanced: [{ frameRate: 50 }, { frameRate: 25 }],
  },
};

const stream = await navigator.mediaDevices.getUserMedia(constraints);

// Here, I asked for a full HD video at 50 fps and if not possible at 25 fps.

Depending on the support of the zoom constraints, you are able to zoom a bit into the video you have.

// Assuming you have captured a video media
const [video] = stream.getVideoTracks();

video.applyConstraints({ advanced: [{ zoom: 120 }] });

// Using my camera, I can zoom from 100 (the farthest) to 140 (the nearest).

Note: Be careful, it seems that the zoom level is kept when calling again getUserMedia call even if the page is refreshed - Chrome 112.

Note: I was not able to use the zoom constraint outside the advanced section.

Constraints error

The APIs getUserMedia and appliConstraints can fail for many reasons and specific error names have been defined. Two of these relate to constraints: OverconstrainedError and TypeError (only for getUserMedia).

OverconstrainedError occurs when at least one mandatory constraint to be satisfied fails. The error returns the name of the first unsatisfied constraint.
TypeError occurs only when the constraints are empty: Either because there is no constraint (no parameter used) or if the constraints for audio and video are both equal to false.

Note: So don’t forget to catch the code.

Interesting new stuff

There is an ongoing W3C working draft called Media Capture and Streams Extensions that defines some potential evolutions. Part of the document addresses the constraints.

New constraint backgroundBlur

When you launch Chrome using the chrome://flags/#enable-experimental-web-platform-features flag, you can test the background blur feature that is built into the browser if your camera supports it.

Once started this way, you can add a new backgroundBlur constraint in the advanced section to enable the feature.

Then it can be modified using the applyConstraints function.

If it is made official, this could be very good news for web developers who don’t want to spend months trying to copy/paste tensorFlow and WASM based background blur implementations and want to keep the code as simple as possible by just adding a new constraint :-) Thanks to the browsers for making this possible!