
## Introduction to Shaders in BabylonJS

A shader is a program processed by the Graphical Processing Unit (GPU) to produce a screen image by manipulating data to
produce individual pixels. The GPU is optimised, through parallel processing, to deal with these thousand of operations
in an extremely fast way.

## Suggested Pre-Reading

-   How To:
    -   [Custom Meshes](/divingDeeper/mesh/creation/custom/custom)
    -   [Updating Vertices](/divingDeeper/mesh/creation/custom/updatingVertices)
-   Resources:
    -   [Normals in BJS](/divingDeeper/mesh/creation/custom/vertexNormals)

## An Overview

### Basic

To produce a BabylonJS scene, code is written in Javascript which the BabylonJS Engine processes and displays the result on screen.
The scene can alter through changes to the meshes, the lights or camera position. To show possible changes in a timely way the screen
display (frame) is re-drawn up to 60 frames per second.

Simplifying, the process is

-   Scene Code is processed in the CPU by the BJS Engine Code to produce a Virtual 3D Model
-   Virtual 3D Model is processed in the CPU by the BJS Engine Code to produce Shader GPU Code
-   Shader GPU Code is processed by GPU to produce screen image.

For example the BabylonJS Engine takes this code

```javascript
var box = BABYLON.MeshBuilder.CreateBox("box", {}, scene);
```

and turns it into vertex data including positions, colors and normals.

The BabylonJS Engine creates the shader code for this data and is passed to the GPU.

### Custom

Much more than this as well as Scene Code you can write your own user Shader Code so that
the process becomes:

-   Scene Code is processed in the CPU by the BJS Engine Code to produce a Virtual 3D Model
-   Virtual 3D Model and User Shader Code is processed in the CPU by the BJS Engine Code to produce the Shader GPU Code
-   Shader GPU Code is processed by GPU to produce the screen image.

## Types of Shader

Shaders are written in Graphics Library Shader Language (GLSL) and come in two parts:

1. Vertex Shader: this takes the data for each vertex and determines where on the screen the pixel for it will be displayed and its color.

2. Fragment Shader: this uses data from the Vertex Shader to determine the position and colour of the pixels to represent each facet of the mesh.

Fragment Shaders are sometimes refered to as Pixel Shaders.

![Shaders](/img/how_to/Shaders/shade1.jpg)

## Passing Variables

The vertex data for position, normal and uv coordinates are passed to the Vertex Shader as variables of category attribute.
User data can be passed to both the Vertex Shader and the Fragment Shader as variables of category uniform.
Data can be passed from the Vertex Shader to the Fragment Shader with variables of category varying.

A vital uniform variable to declare in the Vertex Shader is `worldViewProjection` as the BabylonJS Engine uses this to
pass scene 3D - 2D projection data to the Vertex Shader.

![Pass Variables](/img/how_to/Shaders/shade2.jpg)

## Variable types

All variables used in both shaders must be given a type and any numbers assigned to the variable must be consistent with its type.

For example:

```
int n = 2;
float r = 2.0;
```

The following example with throw an error:

```
float r = 2;
```

Some examples of types are

-   `vec2`: a two dimensional vector of floating-point numbers
-   `vec3`: a three dimensional vector of floating-point numbers
-   `mat4`: a matrix with 4 columns and 4 rows floating-point numbers
-   `sampler2D`: a 2D texture image

Since vertex positions need to be as accurate as possible all floating-point numbers should be set as having high precision.
This is done at the start of the code for each shader using:

```
precision highp float
```

## Built In Variables

The GLSL language has a number of built in variables. Two are vital to the operation of the two shaders and are always necessary:

| Variable     | Description                                                     |
| ------------ | --------------------------------------------------------------- |
| gl_Position  | provide positional data for screen coordinates                  |
| gl_FragColor | provide colour data for the representation of a facet on screen |

## Built In Inputs

| Attribute | Type | Description                                            |
| --------- | ---- | ------------------------------------------------------ |
| position  | vec3 | pixel: vertex position / fragment: face pixel position |
| normal    | vec3 | pixel: vertex normal / fragment: face pixel normal     |
| uv        | vec2 | texture coordinate                                     |

&nbsp;

| Uniform             | Type  | Description                                            |
| ------------------- | ----- | ------------------------------------------------------ |
| world               | mat4  | mesh global transforming state (move + rotate + scale) |
| worldView           | mat4  | global view part of mesh                               |
| worldViewProjection | mat4  | global camera                                          |
| view                | mat4  | mesh local view part                                   |
| projection          | mat4  | local camera                                           |
| time                | float | per each frame                                         |

## Functions

Functions needed to be typed as do their parameters and have the form:

```
float NAME(typed parameters) {
	*code*
}
```

## Running Shader Code

Both the Vertex and the Fragment Shader are run from a function which must be called `main` and be of type `void` since it returns
no result. It must also must type the empty parameter list as void:

```
void main(void) {
	*code*
}
```

![Shader Code](/img/how_to/Shaders/shade3.jpg)

## Putting Shader Code in BabylonJS

Here are four ways of putting shader code into your scene:

1. Use [BabylonJS Create Your Own Shader (CYOS)](https://www.babylonjs.com/cyos/) and download a zip file
2. Write the Vertex and Fragment Shader Code into `<script>` tags
3. Write, save and import a Vertex and Fragment Shader file of type _.fx_ into your code
4. Use the [ShaderBuilder extension](https://github.com/BabylonJS/Extensions/tree/master/ShaderBuilder) of BabylonJS.

More details on this can be found below.

## Further Reading

## How To

-   [Putting Shader Code in BabylonJS](/advanced_topics/introToShaders/shaderCodeInBjs)
-   [ShaderMaterial Object](/advanced_topics/introToShaders/shaderMaterial)



To create the minified version of Babylon.js you just have to 
use [gulp](https://github.com/BabylonJS/Babylon.js/tree/master/Tools/Gulp)

But you may want to work with all basic files. In this case, please download the code and use the following include (be aware to have the babylon folder at the root of your site. If you want to use your own shaders in this case, you just have to reference them with a "./" at beginning):

```html
	<script src="tools/DevLoader/BabylonLoader.js"></script>
```

And add this command to your script:

```javascript
    BABLONDEVTOOLS.Loader
        .load(function() {
            // Your code using Babylon.
        }
    );
```

This will take care of referencing all the required files and launch your code once ready.

If you need any script files to load after Babylon, you can require them:

```javascript
    BABLONDEVTOOLS.Loader
        .require('index.js')
        .load();
```

You can also control the root of your Babylon Folder:

```javascript
    BABLONDEVTOOLS.Loader
        .root('../../')
        .require('index.js')        
        .require('other.js')
        .load();
```

When working with all files, you can provide the following information to the engine:

* BABYLON.Engine.CodeRepository: By default, this value points to "/src/"
* BABYLON.Engine.ShadersRepository: By default, this value points to "/src/Shaders/"


## Compressed Textures for GPU

There are special formats of textures which are optimized for access by graphics processors.  They differ from formats whose primary mission is to hold / transmit image data for use on a CPU.  Examples of image formats are .JPG & .PNG.  Formats catering to GPUs may not be ones you are likely to have heard of.  The file extensions for some of them are also not well established.

## Advantages of Compressed Textures
Unlike image file formats, the data of compressed textures is passed to the graphics hardware in its compressed form.  A .JPG can be very small on disk, but gets expanded by the CPU on its way to the GPU.  Retaining its compressed form is what gives these formats their advantages.  They are:
*  They have a smaller RAM footprint.  This is most helpful for mobile devices which share memory between CPU and graphics cores.  Since textures are multi-dimensional, space requirements can escalate very fast for scenes with many large textures.  Higher-end mobile devices have been able to support up to 4k x 4k textures for some time.  These allow for scenes where the camera can approach things, like a highly detailed object without a repeating pattern, and not look blurry when very close.
*  More compressed image "blocks" can fit into the cache of the processor at any one time.  Caching / data access patterns are very important for high performance computing (HPC), which graphics is a form of.
*  Finally, they could lower battery usage, especially during loading.  Image formats need to be unpacked.  Back when .JPG was being designed, battery computers were mostly just luggables.  The problems being addressed were network speed & disk space.  Who cared then how much CPU was required for re-constituting as long as it was much faster than the network?  Further, mipmaps (ever smaller versions for each power of 2 less in dimensions) need to be generated for image types.  Mipmaps can be built into compressed texture formats.

## The Catch
There is more than one format for compressed textures.  Due to the low level implementation for them in hardware,  support for a given format cannot be added like a software driver update.  Support is manufactured right into the circuitry.  This is less of a  problem when building an iOS, Android, or DirectX targeted application.  For a BJS scene which should ideally be able to run on any device / browser, this is a big problem.  Having separate HTML pages for different devices is not really an acceptable solution.

## The Solution
Starting with Babylon.js v3.0, the compressed texture formats supported by a browser / device can be [detected](http://renderingpipeline.com/webgl-extension-viewer/).  This is done when ```var engine = new BABYLON.Engine(...);``` is encountered. So now your engine instance knows which compressed formats could be used here.  That does not solve that different devices will report different results though.

You can also use this playground: <Playground id="#1SCH7H#5" title="Compressed Textures Example" description="Simple example of multi-platform compressed textures."/> to test which format is supported on your devices.

There is no getting around the fact that you need to provide multiple versions of each texture in different formats(more on that later).  The only way of doing that involves having different files for each of the variants obviously, but the naming structure must be formalized in order to programmatically substitute for the image format file described in a .babylon file.  The image format file of each texture still needs to be on the server as well, in cases when it needs to be used.

### Khronos Texture Container Format,  [.KTX](https://www.khronos.org/opengles/sdk/tools/KTX/) files
Now would be a good time to add the aside that due to the fact that this data is not directly used by CPU's and GPU's do not actually "read" files, there may or may not be an actual native file format for a given compression format.  Even for those that do have an associated file format, writing load code for each separately would be tedious and require support.

Enter compressed texture container files, which can handle multiple or even all texture types.  There are also a few container file formats as well (.DDS, .PVR, & .KTX).  Container files can also have all the mipmaps of a texture inside them.  BJS, implements this feature using KTX container files.  KTX is specifically designed for OpenGL, and forces all the arcane code to handle any format OpenGL supports onto the file encoder / generator, even future formats without us doing anything other than adding extension detection, like ASTC.

Here is a chart of all the current formats possible for WebGL, listed in the order chosen when hardware supports multiple formats  (tie breakers):

|Format|Extension|Description|Alpha Capable
|---|---|---|---
|ASTC|*-astc.ktx|Newly approved for WebGL, most powerful, cross-platform.  Implemented in many newer processors, but not exposed by any browsers yet.| Always
|DXT|*-dxt.ktx|Direct X, available primarily on Desktop Operating Systems.|Yes
|PVRTC|*-pvrtc.ktx|Proprietary.  Power VR chips (includes all Apple iOS processors).  Must be square.|Yes
|ETC2|*-etc2.ktx|ETC1 + alpha capable.  Required by WebGL 2 (or at least OpenGL ES 3, on which WebGL 2 is based).|Yes
|ATC|*-atc.ktx|Format originating at AMD.  No encoder which supports .KTX found at this time.|Yes
|ETC1|*etc1.ktx|Wide support among older mobile devices.  Need to fall back to images for .PNG files.|No

Once your engine instance is established, you need to indicate the compressed formats that you have put on the server from which it can pick from.  This should probably be done very early, as follows:

```javascript
// order & case do not matter
var available = ['-astc.ktx', '-dxt.ktx', '-pvrtc.ktx', '-etc2.ktx', '-etc1.ktx'];
var formatUsed = engine.setTextureFormatToUse(available);
```

## Generation using [PVRTexTool](https://community.imgtec.com/developers/powervr/tools/pvrtextool/)
The are multiple encoder programs for .KTX files (see .KTX link above).  Most also provide for batch processing, since many formats are very CPU intense.  PVRTexTool has been narrowed down for providing additional support.  It has a lot going for it, including both a GUI & command line interface for Windows, OSX, and Linux.  It is also the only encoder which does PVRTC format, which is needed on iOS.

If you are going to do the encoding on your own in the PVRTexTool GUI tool, there are a few things to keep in mind:
*  The texture must be encoded with the Y-axis flipped. ('Vertical Flipped' checkbox at bottom of Encode window)
*  PVRTC textures must be square for iOS.  ('Make Square' button in the Toolbox window using Resize tool)
*  ASTC support is dropped in from an [external program](https://github.com/ARM-software/astc-encoder) from ARM that you put on the path.  (See PVRTexTool [manual](http://cdn.imgtec.com/sdk-documentation/PVRTexTool.User+Manual.pdf), section 1.2.1)
*  DXT support is done with 'BCx' names.  (Better UI design for WebGL has been [requested](https://community.imgtec.com/forums/topic/could-webgl-be-added-as-an-encoding-groupapi/).)
*  Indicating to generate mipmaps should really be done as well. (checkbox to the left of 'Vertical Flipped')
*  Always use a Linear-RGB encoding type.
*  Not sure a power of 2 size is absolutely required, but scripts below size up to the next power of 2 size.

### DOS batch scripts for PVRTexTool
There are 2 batch scripts in the [BJS repo](https://github.com/BabylonJS/Babylon.js/tree/master/Tools/CompressedTextured).  They both require that PVRTexToolCLI.exe, be put on the execution path just as the ASTC drop-in was.  Doing both at the same time, and locating these 2 .BAT files in the same place seems like a good thing to do.

#### make-ktx-batch.bat
This script goes through the current directory, and writes a ```ktx-batch.bat``` file there.  When you then call ```ktx-batch.bat```, it will make an ASTC, DXT, PVRTC, ETC2, and ETC1 file for each .JPG and .PNG in the directory. Note that a .PNG extension is an indicator to use an alpha capable sub-type.  ETC1 does not support alpha, so the .PNG will be used as a fall back, if ETC1 ends up being chosen.

There is a single argument which indicates the quality of the texture. Specify D for developer level, Q for production quality.  Tip: unless you are testing if this fixes hanging issues on mobile devices, you could just delay enabling this till all your textures get finalized.  Also, do not rely on the results you get from using D on a desktop.  DXT does not really have variable quality.

#### ktx-files.bat
This script will create the 5 variations of a image file.  It can run for a very long time for Q setting.  The ASTC type will use 100% of all your cores, so your system can be pretty unusable.  Good to kick it off at the end of day.  Also, due to running time, it will skip any files already existing.  To re-do files, delete the existing versions first.

To recap (in a command shell):
```dos
cd my-directory-with-images
make-ktx-batch Q
ktx-batch
```

### Node.js script for PVRTexTool
This is a script that generates PVRTC, ETC1, ETC2, ASTC textures from png and jpg files. It can run on node or as a gulp task. Also it can be configured to generate all texture types or specific ones. More information about how to install and configure the script can be found [here](https://www.npmjs.com/package/babylonjs-texture-generator).

### Basis file format

Another way to store compressed image textures is throught the .basis file format

See: https://github.com/BinomialLLC/basis_universal

Basis Universal is a "supercompressed" GPU texture and texture video compression system that outputs a highly compressed intermediate file format (.basis) that can be quickly transcoded to a wide variety of GPU texture compression formats: PVRTC1 4bpp RGB, BC7 mode 6 RGB, BC1-5, ETC1, and ETC2. We will be adding ASTC RGB or RGBA, BC7 mode 4/5 RGBA, and PVRTC1 4bpp RGBA next. Basis files support non-uniform texture arrays, so cubemaps, volume textures, texture arrays, mipmap levels, video sequences, or arbitrary texture "tiles" can be stored in a single file. The compressor is able to exploit color and pattern correlations across the entire file, so multiple images with mipmaps can be stored very efficiently in a single file.

Basic example: <Playground id="#4RN0VF" title="Basic Example of Basis Textures" description="Simple example of using .basis compressed textures in your scene."/>
Basis vs png example scene: <Playground id="#E4VDDW" title="Basis Texture vs. png Texture" description="Simple example of .png vs basis compressed textures."/>


Starting with Babylon.js v3.1, we introduced the NullEngine which is a version of the main Babylon.js engine but with no need for a webgl capable device.

The NullEngine will not produce any rendering and thus can be used in a node / server side environment.

It can be used to:
- Run tests
- Run a server side version of your application / game
- Use specific Babylon.js services (like gltf loaders for instance)

## General

To use a headless version of Babylon.js, just run:

```
var engine = new BABYLON.NullEngine();
var scene = new BABYLON.Scene(engine);
```

If you want to use it in a node.js environment, you may need to add some additional resources (like support for xhr):

```
var BABYLON = require("../../dist/preview release/babylon.max");
var LOADERS = require("../../dist/preview release/loaders/babylonjs.loaders");
global.XMLHttpRequest = require('xhr2').XMLHttpRequest;

var engine = new BABYLON.NullEngine();
var scene = new BABYLON.Scene(engine);

var light = new BABYLON.PointLight("Omni", new BABYLON.Vector3(20, 20, 100), scene);

var camera = new BABYLON.ArcRotateCamera("Camera", 0, 0.8, 100, BABYLON.Vector3.Zero(), scene);

BABYLON.SceneLoader.ImportMesh("", "https://playground.babylonjs.com/scenes/", "skull.babylon", scene, function (newMeshes) {
    camera.target = newMeshes[0];

    console.log("Meshes loaded from babylon file: " + newMeshes.length);
    for (var index = 0; index < newMeshes.length; index++) {
        console.log(newMeshes[index].toString());
    }

    BABYLON.SceneLoader.ImportMesh("", "https://www.babylonjs.com/Assets/DamagedHelmet/glTF/", "DamagedHelmet.gltf", scene, function (meshes) {
        console.log("Meshes loaded from gltf file: " + meshes.length);
        for (var index = 0; index < meshes.length; index++) {
            console.log(meshes[index].toString());
        }
    });

    console.log("render started")
    engine.runRenderLoop(function() {
        scene.render();
    })
});

```

## Configuration

You can specifiy some options to the NullEngine constructor:

```
var engine = new BABYLON.NullEngine({
    renderWidth: 512,
    renderHeight: 256,
    textureSize: 512
});
```

With `renderWidth` and `renderHeight` you can define the size of the virtual canvas used to simulate the render surface.
With `textureSize`, you can define the size of all textures (which will be all black) used by the NullEngine.

## Troubleshooting

While the goal of the NullEngine is to completely replace the Engine class, some features cannot be used server side:
* camera.attachControl cannot be used as it requires a HTML eLement
* DynamicTexture cannot be used as they rely on HTML canvas

Furthermore as we already mentioned earlier, you need to provide an implementation of the XmlHttpRequest class.


## Introduction

Starting with v3.0, Babylon.js supports rendering using WebGL1 and WebGL2 contexts.
The support is transparent for developers. By default the engine tries to get a WebGL2 context. If none is available then a WebGL1 one is retrieved.

You can test which version of WebGL is enabled with: `engine.webGLVersion` property.

## Shaders

When WebGL2 is enabled, the shaders are automatically converted to GLSL v3.0. Babylon.js will then automatically take advantage of extended instruction/uniform counts.

If you are using custom shaders, the best idea would be to provide GLSL v2.0 shaders. This way your code will work on both contexts.
You can obviously provide only v3.0 shaders but in this case your code will only work when WebGL2 is enabled.

## Supported features

You can find here the list of supported features and the backward compatibility options (when available)

| Feature                      | Description                                                                                                                                                                                                                                        | WebGL1 compatibility                                                                                          | Demo                                                                                                             | More info                                                                |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Depth Frag                   | Used to compute logarithmic depth buffer                                                                                                                                                                                                           | Yes through an [extension](https://www.khronos.org/registry/webgl/extensions/EXT_frag_depth/)                 | <Playground id="#1180R5#15" title="Depth Frag" description="Depth Frag"/>                               | [Documentation](/divingDeeper/materials/advanced/logarithmicDepthBuffer) |
| Multisample render targets   | Rendertarget textures can be multisampled to get antialiasing effect                                                                                                                                                                               | No. Has no effect on WebGL1 context                                                                           | <Playground id="#12MKMN" title="Multisample Render Targets" description="Multisample render targets."/> | [See below](/advanced_topics/webGL2#multisample-render-targets)          |
| Standard derivatives         | Standard derivatites are used in Babylon.js to help compute realtime bump                                                                                                                                                                          | Yes through an [extension](https://www.khronos.org/registry/webgl/extensions/OES_standard_derivatives)        | [Demo](https://www.babylonjs.com/Demos/Bump/)                                                                    | [Documentation](/divingDeeper/materials/using/moreMaterials)             |
| Texture LOD                  | Used by PRBMaterial to simulate microsurface                                                                                                                                                                                                       | Yes through an [extension](https://www.khronos.org/registry/OpenGL/extensions/EXT/EXT_shader_texture_lod.txt) | [Demo](/divingDeeper/materials/using/HDREnvironment)                                                             | [Documentation](/divingDeeper/materials/using/introToPBR)                |
| Vertex array objects (VAO)   | A Vertex Array Object (or VAO) is an object that describes how the vertex attributes are stored in a Vertex Buffer Object (or VBO)                                                                                                                 | Yes through an [extension](https://www.khronos.org/registry/webgl/extensions/OES_vertex_array_object/)        | N/A. Every rendering is done with VAO by default                                                                 | [See below](/advanced_topics/webGL2#vertex-array-objects)                |
| Uniform buffer objects (UBO) | An uniform buffer object (or UBO) let you specify a group of uniforms from a buffer                                                                                                                                                                | No. Uniforms are handled independently on WebGL1 context                                                      | N/A. Materials supporting UBO automatically uses them                                                            | [See below](/advanced_topics/webGL2#uniform-buffer-objects)              |
| Multiple Render Target (MRT) | Several Render Targets can be rendered in the same draw call.                                                                                                                                                                                      | Yes through an [extension](https://www.khronos.org/registry/webgl/extensions/WEBGL_draw_buffers)              | <Playground id="#NZ6P07" title="Multiple Render Target" description="Multiple Render Target"/>          | [See below](/advanced_topics/webGL2#multiple-render-target)              |
| Occlusion Queries            | Occlusion queries detect whether a Mesh is visible in the current scene or not                                                                                                                                                                     | Yes through an [extension](https://www.khronos.org/opengl/wiki/Query_Object#Occlusion_queries)                | <Playground id="#QDAZ80#3" title="Occlusion Queries" description="Occlusion Queries"/>                  | [See below](/advanced_topics/webGL2#occlusion-queries)                   |
| 3D Textures                  | 3D textures are textures with a 3rd dimension. You can see them as multiple 2D textures where every texture is a slice in the 3d texture.                                                                                                          | No. Cannot be created in WebGL1                                                                               | This feature will automatically be used when possible.                                                           | [See below](/advanced_topics/webGL2#3d-textures)                         |
| 2D Array Textures            | 2D array textures are very similar to 3D textures but are designed for constructing a texture atlas instead of a volumetric texture.                                                                                                               | No. Cannot be created in WebGL1                                                                               | <Playground id="#XEVUD9" title="2D Array Textures" description="2D Array Textures"/>                    | [See below](/advanced_topics/webGL2#2d-array-textures)                   |
| Power of two textures        | In the past, to achieve the best performance and higher quality texture rendering, images with dimensions that are a power of two were required. With support for WebGL2 this is no longer the case, any sized texture will be rendered optimally. | Yes, however Babylon will resize textures to be a power of two causing a hit to performance                   | N/A. This is done by default                                                                                     | [See below](/advanced_topics/webGL2#power-of-two-textures)               |
| Transform feedback buffer    | Transform feedback buffer can be used to update vertex buffers from GPU. Babylon.js uses it to implement GPU particles                                                                                                                             | No. Not supported on WebGL1                                                                                   | <Playground id="#PU4WYI" title="Power Of Two Textures" description="Power Of Two Textures"/>            | [See particles documentation](/divingDeeper/particles/particle_system/particle_system_intro#gpu-particles)       |
| Shadow Samplers              | Shadow samplers are used to enable PCF depth comparison on the hardware. Babylon.js uses it to implement PCF and PCSS shadows.                                                                                                                     | No. Not supported on WebGL1 (shadows fall back to poisson sampling)                                           | <Playground id="#ZT8BKT#1" title="Shadow Samplers" description="Shadow Samplers"/>                      | [See shadows documentation](/divingDeeper/lights/shadows)                         |
| More precise shadows         | Shadow maps can now use 32 bits depth buffers improving by a large scale the precision of the shadows.                                                                                                                                             | No. Not supported on WebGL1 (shadows precision will fall back to 16 bits)                                     | <Playground id="#ZT8BKT#1" title="More Precise Shadows" description="More Precise Shadows"/>            | [See shadows documentation](/divingDeeper/lights/shadows)                         |

## Multisample render targets

By default render targets (like mirrors for instance) are created without support for multisampling. To turn it on, just define a value for `renderTarget.samples` > 1.
On WebGL1 context, this will do nothing. On WebGL2 context, this will enable multisampling (more samples imply better antialiasing but a slower rendering).

Here is an example of a mirror (512x512) with and without multisampling:

| No MSAA (1 sample)                 | MSAA (8 samples)                 |
| ---------------------------------- | -------------------------------- |
| ![Title](/img/features/nomsaa.jpg) | ![Title](/img/features/msaa.jpg) |

## Vertex array objects

When possible (either on WebGL2 context or when extension is available on WebGL1 context), Babylon.js will use VAO to control rendering. VAO are a kind of geometry objects. Instead of sending all attributes and buffers used by a mesh (one for position, one for normal, one for indices, one for texture coordinates, etc..), you can build a VAO which will keep track of all attributes / buffers used.

At rendering time, you just have to define one VAO instead of multiple VBO (vertex buffer object).

You can find more details on [Tojicode's blog](http://blog.tojicode.com/2012/10/oesvertexarrayobject-extension.html).

## Uniform buffer objets

On WebGL1 context all uniforms are sent to GPU independently. This means that if your shader uses 16 matrices, you will call WebGL API 16 times to update all matrices before using your shader.

On WebGL2 context, you can use a UBO to set the values in a typed array all inside JavaScript. This means that it's much faster. When all the values are set you can then send them to the GPU with only one call.

You can find more details on [WebGL 2 specification](https://www.khronos.org/registry/webgl/specs/latest/2.0/#3.7.16)

## Multiple Render Target

On former WebGL1, one draw call meant 1 target texture. Now you can bind several target textures to a shader and specify inside the fragment shader the colors you want to put on each texture. Essentially it saves you a lot of CPU time and you can achieve advanced effects like [Deferred Shading](https://fr.wikipedia.org/wiki/Deferred_Shading).

In Babylon.js, our first use of this technique is to render a geometry buffer of the scene.

## Occlusion queries

Occlusion queries detect whether a Mesh is visible in the current scene or not, and based on that the Mesh get drawn or not. Occlusion queries is useful when you have an expensive object on the scene and you want to make sure that it will get drawn if it is visible to the camera and it is not behind any opaque object. BabylonJs provides an implementation for Occlusion queries using property occlusionType in AbstractMesh Class

[Babylon.js Occlusion Queries Feature](/divingDeeper/occlusionQueries)

[WebGL 2 Occlusion Queries](https://www.khronos.org/opengl/wiki/Query_Object#Occlusion_queries)

## 3D textures

3D textures are mostly used for volumetric effects like color grading, fire, smoke, etc. WebGL 2 support for 3D textures is as good as that for 2D textures.

So far Babylon.js will use them for color grading texture: <Playground id="#17VHYI#2" title="3D Textures Example" description="Simple example of using 3D textures."/>

## 2D array textures

2D array textures allow you to pass a texture atlas to a custom shader. This could be used whenever you have multiple, distinct, 2D textures that you want to blend or switch between inside your shader. For example tiles, terrain splatting or frames of an animation. Using array textures ensures that distinct layers are sampled _as if_ they were separate textures, so there will be no bleeding between different sections of the atlas.

Usage is very similar to 3D textures: create a `RawTexture2DArray`, use `setTexture` on the shader material, and use a sampler of type `sampler2DArray`. Sample using `texture(yourSampler, vec3(u,v,layerIndex))` where `layerIndex` is a 0-based index into the array.

More information is available on the [Khronos wiki](https://khronos.org/opengl/wiki/Array_Texture).

<Playground id="#XEVUD9" title="2D Array Textures Example" description="2D Array Textures Example"/>

## Power of two textures

On WebGL1 context, all textures are resized to a power of two to produce the best quality. This resize may impact performance.

On WebGL2 context, no resize is required and any size texture will be rendered with the best quality.

You can find more details on [WebGL 2 specification](https://www.khronos.org/registry/webgl/specs/latest/2.0/#4.1.3)



## Web GPU

## Introduction
As you might be aware, the next gen Web 3D is starting to rise.

Following the specification from the [W3C group](https://github.com/gpuweb/gpuweb), we have started our journey to support this new tech. Our plan is to deliver a support with WebGL feature parity and  the new WebGPU specifics in 5.0.

![Inspector](/img/extensions/WebGPU.png)

## Why ?
The promise behind WebGPU is an awesomely faster API providing lower level control to the graphic resources from Javascript. We hope to bring those extra performances to you in order to create even bigger Web 3D experiences through an API you are already familiar with: Babylon.js.

Also this is brigging new cool toys to the table like Compute Shaders and more to come.

## Where are we ?

See [the dedicated page](/advanced_topics/webGPU/webGPUStatus).

You can also follow our progress on out [Github dedicated issue](https://github.com/BabylonJS/Babylon.js/issues/6443).

## How ?
The current implementation is now merged in the main branch of the repo [Github](https://github.com/BabylonJS/Babylon.js).

We are putting a big effort all along our next release to implement the support and every contribution is more than welcome; So feel free to create some PR if you are interested in contributing to a fresh engine implementation based on cutting edge tech.

## What could go wrong ?
Even if we are now well in the development process, we are subject to potential changes as well as spec uncertainties. We will deal with them one at a time but we may be subject to change the release schedule at any time.

We have the full support from the amazing teams implementing the browsers support so it will be easier to directly chat with them when needed.

## What would the migration look like for my app ?
As back compat is one of our pillar, the only difference we wish to have is the engine initialization which needs to be asynchronous:

```javascript
const engine = new BABYLON.WebGPUEngine(canvas);
await engine.initAsync();
```

## Will WebGL still be supported ?
FOR SURE :-) There is no change of plans and we will deliver both supports side by side.

## I want to test it
You can refer [to this page](https://github.com/gpuweb/gpuweb/wiki/Implementation-Status) for detailed information on browser support.

Assuming you're using a browser that supports WebGPU, you can try it out yourself in the The [Playground](https://playground.babylonjs.com/) 

![WebGPU](/img/extensions/webGPUPlayground.jpg)

You can follow the status from the dedicated [Chrome Status Platform Page](https://www.chromestatus.com/feature/6213121689518080).

All the demos code is available on [Github](https://github.com/BabylonJS/Website/tree/master/build/Demos/WebGPU) so that you could compare both the webgl and webgpu versions and notice there are currently no differences beside the initialization. We will try to keep it this way :-)


## A bit about augmented reality

The idea behind augmented reality is simple - show the real world, but add information on top of it. As opposed to virtual reality where you are fully immersed in a different scene and have no actual contact to the real world, augmented reality lets you interact with it.

## Getting started with Augmented reality

### WebXR and AR

Augmented reality using Babylon.js will be heavily using WebXR, so I will recommend you to first start with the [getting started with WebXR](/divingDeeper/webXR/introToWebXR) guide. Most of the information that is valid for immersive VR sessions is also valid for immersive AR sessions. The few main differences between the two will be explained here.

### Supported devices

Immersive AR sessions are (currently) supported on two types of devices - mobile phones and firefox reality on the Hololens.

Mobile phones using android support immersive AR sessions on chrome (stable/canary). Note that you will need to install [AR Core](https://play.google.com/store/apps/details?id=com.google.ar.core), otherwise it will be a very short experience.

Hololens 2 supports WebXR and immersive AR sessions when using [Firefox Reality for hololens](https://www.microsoft.com/en-ca/p/firefox-reality/9npq78m7nb0r).

To check your scene on a desktop you can use the WebXR emulator, which supports a set of features of AR and lets you enter an immersive AR session when choosing the mobile mode.

### Simple scene in immersive AR

The simplest immersive AR sample is a port of an immersive VR scene:

```javascript
var createScene = async function () {
  var scene = new BABYLON.Scene(engine);
  var camera = new BABYLON.FreeCamera(
    "camera1",
    new BABYLON.Vector3(0, 5, -10),
    scene
  );
  camera.setTarget(BABYLON.Vector3.Zero());
  camera.attachControl(canvas, true);
  var light = new BABYLON.HemisphericLight(
    "light1",
    new BABYLON.Vector3(0, 1, 0),
    scene
  );
  light.intensity = 0.7;
  var sphere = BABYLON.Mesh.CreateSphere("sphere1", 16, 2, scene);
  sphere.position.y = 2;
  sphere.position.z = 5;

  const xr = await scene.createDefaultXRExperienceAsync({
    // ask for an ar-session
    uiOptions: {
      sessionMode: "immersive-ar",
    },
  });

  return scene;
};
```

Playground for a simple immersive AR scene: <Playground id="#F41V6N#32" title="Simple Immersive AR Scene" description="Simple example of an immersive AR scene."/>

Notice that no environment was created. As opposed to immersive VR sessions, AR doesn't require a skybox or ground. If you want to define ground and remove it when entering AR (for example if you develop an experience for both desktop and AR) you can use the background remover feature, defined later in this page.

## Features

Some features require the Incubation flag in the latest chrome canary. Visit chrome://flags/ and enable WebXR incubation.

You will then need to ask WebXR to enable this feature. this can be done using the optionalFeature parameter in the default experience helper:

```javascript
const xr = await scene.createDefaultXRExperienceAsync({
  uiOptions: {
    sessionMode: "immersive-ar",
  },
  optionalFeatures: true,
});
```

This will enable all optional features that we support. You can also add individual features by using an array insteaf of a boolean:

```javascript
const xr = await scene.createDefaultXRExperienceAsync({
  uiOptions: {
    sessionMode: "immersive-ar",
  },
  optionalFeatures: ["hit-test", "anchors"],
});
```

### Hit test

Hit-test is used for sending a ray into the real world and receiving information about intersection in space. You can read about it in the [hit test w3c draft](https://immersive-web.github.io/hit-test/). Think about a ray that is broadcasted from your phone's screen towards to object you are looking for. If the device's AR capabilities allows it, it will let you know the position and orientation of the point relative to you.

To enable hit-testing, add this after initializing XR

```javascript
// featuresManager from the base webxr experience helper
const hitTest = featuresManager.enableFeature(BABYLON.WebXRHitTest, "latest");
```

In typescript you can also get the type set correctly:

```javascript
// featuresManager from the base webxr experience helper
const hitTest = featuresManager.enableFeature(BABYLON.WebXRHitTest, 'latest') as BABYLON.WebXRHitTest;
```

This will enable the default behavior of hit-testing which is sending a single hit-test ray from the center of the display forward on each frame.

The options that you can pass are descriped in this interface:

```javascript
export interface IWebXRHitTestOptions {
  /**
   * Do not create a permanent hit test. Will usually be used when only
   * transient inputs are needed.
   */
  disablePermanentHitTest?: boolean;
  /**
   * Enable transient (for example touch-based) hit test inspections
   */
  enableTransientHitTest?: boolean;
  /**
   * Offset ray for the permanent hit test
   */
  offsetRay?: Vector3;
  /**
   * Offset ray for the transient hit test
   */
  transientOffsetRay?: Vector3;
  /**
   * Instead of using viewer space for hit tests, use the reference space defined in the session manager
   */
  useReferenceSpace?: boolean;
}
```

`disablePermanentHitTest` will disable the constant hit-testing, while `enableTransientHitTest` will enable hit-tests when touching the screen. The offsets define the offset from which the ray will be sent (relative to the center of the device's view).

After enabling the feature, you can register to get updates using the `onHitTestResultObservable`:

```javascript
// a dot to show in the found position
const dot = BABYLON.SphereBuilder.CreateSphere(
  "dot",
  {
    diameter: 0.05,
  },
  scene
);
dot.isVisible = false;
hitTest.onHitTestResultObservable.add((results) => {
  if (results.length) {
    dot.isVisible = true;
    results[0].transformationMatrix.decompose(
      dot.scaling,
      dot.rotationQuaternion,
      dot.position
    );
  } else {
    dot.isVisible = false;
  }
});
```

This will show the dot only if hit-test worked, and will hide it if it didn't. The dot will be projected to the real world using the information provided by the system.

A simple example for WebXR hit-test using Babylon.js: <Playground id="#XWBES1" title="WebXR Hit-Test Using Babylon.js" description="Simple example of WebXR hit testing."/>. Open this with your AR device (probably your android smartphone) and point the device at a textured plane (like your floor or door). The marker will be shown at the right location if/when the system scanned the plane correctly. 

### Anchors

Anchors are tracked points in space that the system will constantly update as you continue scanning your environment. The points' transformation will be constantly updated by the underlying system. You can read more about anchors at the [WebXR anchors module](https://immersive-web.github.io/anchors/) w3c proposal.

Enable the anchors system using:

```javascript
// featuresManager from the base webxr experience helper
const anchorSystem = featuresManager.enableFeature(BABYLON.WebXRAnchorSystem, 'latest');
```

or for typescript:

```javascript
// featuresManager from the base webxr experience helper
const anchorSystem = featuresManager.enableFeature(BABYLON.WebXRAnchorSystem, 'latest') as BABYLON.WebXRAnchorSystem;
```

Some options can be passed to the anchor system:

```javascript
export interface IWebXRAnchorSystemOptions {
    /**
     * a node that will be used to convert local to world coordinates
     */
    worldParentNode?: TransformNode;

    /**
     * If set to true a reference of the created anchors will be kept until the next session starts
     * If not defined, anchors will be removed from the array when the feature is detached or the session ended.
     */
    doNotRemoveAnchorsOnSessionEnded?: boolean;
}
```

Anchors are removed when exiting the XR session (note that the anchors are removed, and not meshed that are attached to them), which is the recommended behavior, as anchors cannot be referenced between sessions.

If you want to prevent that from happening, use the `doNotRemoveAnchorsOnSessionEnded` when initializing the anchor system:

```javascript
const anchorSystem = featuresManager.enableFeature(BABYLON.WebXRAnchorSystem, 'latest', { doNotRemoveAnchorsOnSessionEnded: true });
```

The anchor system fits perfectly to the hit-test feature if you want to add an anchor at a hit test position. To do that use the `addAnchorPointUsingHitTestResultAsync` function:

```javascript
const arTestResult = getMeTheResultINeed();
const anchorPromise = anchorSystem.addAnchorPointUsingHitTestResultAsync(lastHitTest);
```

To add an anchor in any position and rotation in the scene, use the `addAnchorAtPositionAndRotationAsync` function:

```javascript
const { position, rotationQuaternion } = anyRandomMesh;
const anchorPromise = anchorSystem.addAnchorAtPositionAndRotationAsync(position, rotationQuaternion);
```

Note that `anchorPromise` will return a *native XRAnchor* when fulfilled. This will provide you with what the browser returns. To work with anchors **in the babylon way**, we use the observables defined in the anchor module:

```javascript
anchorSystem.onAnchorAddedObservable.add((anchor) => {
    // ... do what you want with the anchor after it was added
})

anchorSystem.onAnchorRemovedObservable.add((anchor) => {
    // ... do what you want with the anchor after it was removed
})

anchorSystem.onAnchorUpdatedObservable.add((anchor) => {
    // ... do what you want with the anchor after it was updated
})
```

The anchor is of type `IWebXRAnchor`:

```javascript
export interface IWebXRAnchor {
    /**
     * A babylon-assigned ID for this anchor
     */
    id: number;
    /**
     * Transformation matrix to apply to an object attached to this anchor
     */
    transformationMatrix: Matrix;
    /**
     * The native anchor object
     */
    xrAnchor: XRAnchor;

    /**
     * if defined, this object will be constantly updated by the anchor's position and rotation
     */
    attachedNode?: TransformNode;
}
```

To attach an anchor to a node (for example a mesh that you want to always be in this location in the scene) use the `attachedNode` variable. This will also update the mesh's transformation when the anchor is updated:

```javascript
const mesh = //...

anchorSystem.onAnchorAddedObservable.add((anchor) => {
    anchor.attachedNode = mesh;
})
```

The mesh will now be tracked by the system and will be located at the requested point. 

You might ask yourself why use the anchor system with hit-test results, as hit-test results are returned by the system with a position defined by the device. Setting the mesh at the hit-test's location will work just fine. The difference is that the system might update the information it has about this position - maybe it found out the plane is at a different transformation, maybe it updated its position in space. Using the anchor system will keep the transformation updated even when the system updated its knowledge of the space.

### Plane detection

Your device is (usually) capable of detecting plane geometries in the real world. To read more about plane detection go to the [Plane detection explainer](https://github.com/immersive-web/real-world-geometry/blob/master/plane-detection-explainer.md). 

Babylon has an experimental plane detection module that works with the underlying system. To enable it:

```javascript
// featuresManager from the base webxr experience helper
const planeDetector = featuresManager.enableFeature(BABYLON.WebXRPlaneDetector, "latest");
```

Just like any module you can configure it using the options object, which is of this type:

```javascript
export interface IWebXRPlaneDetectorOptions {
    /**
     * The node to use to transform the local results to world coordinates
     */
    worldParentNode?: TransformNode;

    /**
     * If set to true a reference of the created planes will be kept until the next session starts
     * If not defined, planes will be removed from the array when the feature is detached or the session ended.
     */
    doNotRemovePlanesOnSessionEnded?: boolean;
}
```

Similar to the anchor system, planes do not stay between sessions. If you want to keep the native XRPlane objects, set the `doNotRemovePlanesOnSessionEnded` to true, and babylon will not delete them.

the plane detector works automatically and offers three observables for you to use:

```javascript
planeDetector.onPlaneAddedObservable.add((plane) => {
    // ... do what you want with the plane after it was added
})

planeDetector.onPlaneRemovedObservable.add((plane) => {
    // ... do what you want with the plane after it was removed
})

planeDetector.onPlaneUpdatedObservable.add((plane) => {
    // ... do what you want with the plane after it was updated
})
```

The plane object is of type IWebXRPlane:

```javascript
export interface IWebXRPlane {
    /**
     * a babylon-assigned ID for this polygon
     */
    id: number;
    /**
     * an array of vector3 points in babylon space. right/left hand system is taken into account.
     */
    polygonDefinition: Array<Vector3>;
    /**
     * A transformation matrix to apply on the mesh that will be built using the polygonDefinition
     * Local vs. World are decided if worldParentNode was provided or not in the options when constructing the module
     */
    transformationMatrix: Matrix;
    /**
     * the native xr-plane object
     */
    xrPlane: XRPlane;
}
```

To create a Babylon polygon from the plane information do the following:

```javascript
const plane = // a reference to an added plane
// add the starting point, so the polygon will close
plane.polygonDefinition.push(plane.polygonDefinition[0]);
// create a polygon mesh builder for the polygons returned from the system
var polygon_triangulation = new BABYLON.PolygonMeshBuilder("name", plane.polygonDefinition.map((p) => new BABYLON.Vector2(p.x, p.z)), scene);
// build the plane with specific thickness
var polygon = polygon_triangulation.build(false, 0.01);

```

A simple usecase for planes is showing them in your scene using polygons. An example for that can be found at the WebXR Plane Detection demo: <Playground id="#98TM63" title="WebXR Plane Dedication Demo" description="WebXR Plane Dedication Demo" image="/img/playgroundsAndNMEs/vrglasses.png"/>


### Background remover

When in AR you want to avoid environment meshes like a skybox and a ground (unless it was your goal to keep them). If you are creating a scene that should work on both regular devices and in AR, you will want the ability to disable certain meshes when entering AR, and re-enabling them when leaving AR. This module does exactly that. It receives a list of meshes, and disables/enables them when needed.

When using the babylon environment helper, the module can do the work for you automatically. In that case, the skybox and ground will be automatically removed if you enabled the feature. 

To enable it use:

```javascript
const xrBackgroundRemover = featuresManager.enableFeature(BABYLON.WebXRBackgroundRemover);
```

To customize the way the module works, use the following configuration object:

```javascript
export interface IWebXRBackgroundRemoverOptions {
    /**
     * Further background meshes to disable when entering AR
     */
    backgroundMeshes?: AbstractMesh[];
    /**
     * flags to configure the removal of the environment helper.
     * If not set, the entire background will be removed. If set, flags should be set as well.
     */
    environmentHelperRemovalFlags?: {
        /**
         * Should the skybox be removed (default false)
         */
        skyBox?: boolean;
        /**
         * Should the ground be removed (default false)
         */
        ground?: boolean;
    };
    /**
     * don't disable the environment helper
     */
    ignoreEnvironmentHelper?: boolean;
}
```

For example if you want the module to remove only the skybox and not the ground, when using the environment helper, enable the feature this way:

```javascript
const xrBackgroundRemover = featuresManager.enableFeature(BABYLON.WebXRBackgroundRemover, 'latest', {
  environmentHelperRemovalFlags: {
    skyBox: true,
    ground: false
  }
});
```

# Demos

<Playground id="#GG06BQ#3" title="XR Measurement Tape" description="XR Measurement Tape Demo" image="/img/playgroundsAndNMEs/vrglasses.png"/>
<Playground id="#KDWCZY" title="Placing A Mesh In Space" description="Simple example of placing a mesh in space." image="/img/playgroundsAndNMEs/vrglasses.png"/>

Advanced Topics

Coming next

The .babylon File Format

Dive deep into the inner workings of the .babylon file format.

Introduction to Shaders

Dive deep into the inner workings of shaders in Babylon.js.

Creating The Mini-fied Version of Babylon.js

Learn how to create the mini-fied version of Babylon.js.

Multi-Platform Compressed Textures

Learn all about multi-platform compressed textures in Babylon.js.

WebGL2 Support

Dive into understanding the full scope of support for WebGL2 in Babylon.js.

WebGPU Support

Dive into understanding the full scope of support for WebGPU in Babylon.js.

Running Babylon.js On A Server

Learn how to run Babylon.js on a server.