Transformers.js Revolutionizes Multimodal AI in the Browser

⚡

Key Takeaways

1Transformers.js enables image classification, captioning, and voice transcription directly in the browser, without a server.

2OpenAI's Vision Transformer and Whisper models are used for multimodal tasks, operating locally on the user's device.

3A local web server is required to bypass browser security restrictions when importing ES modules.

💡Why it matters — This approach enhances user privacy by processing data locally, without transferring it to external servers.

Introduction to Multimodal AI in the Browser

The integration of multimodal artificial intelligence directly into the browser is now possible thanks to Transformers.js. This innovative framework allows for the execution of complex tasks such as image classification, image captioning, and speech transcription, without the need for an external server or API key. Data is processed locally, ensuring enhanced privacy for the user.

Supported Features

Transformers.js offers a wide range of features, including computer vision with image classification, object detection, and segmentation. In terms of audio, it supports automatic speech recognition, audio classification, and text-to-speech synthesis. All these multimodal tasks operate locally in the browser, meaning that data never leaves the user's device.

Implementing Multimodal Capabilities

The tutorial provided by Transformers.js guides users through the creation of three distinct capabilities: image classification, image captioning, and speech transcription. Each capability is implemented in a standalone HTML file, which can be opened directly in a browser. The final section of the tutorial demonstrates how to combine these three capabilities into a single multimodal media analyzer.

System Requirements

To use Transformers.js, a modern browser is required. The required versions include Chrome 109+, Edge 109+, or Firefox 90+, which support ES modules and WebAssembly, essential for the operation of Transformers.js. Additionally, a local web server is needed to bypass browser security restrictions that block ES module imports from file:// URLs. Users can choose from several options to start a local server, such as Python, Node.js, or the Live Server extension for VS Code.

Starting a Local Server

To launch a local server, users can use Python, which is pre-installed on macOS and most Linux systems, with the following command:

python3 -m http.server 8080

For those who prefer Node.js, the following command can be used:

npx serve .

Finally, for VS Code users, the Live Server extension can be installed. Simply right-click on any HTML file and choose "Open with Live Server." Once the server is running, just open http://localhost:8080 in the browser.

Project Structure

The project is structured so that each task has its own HTML file. Here’s how to organize the files in a folder named multimodal-demo:

multimodal-demo/
├── image-classifier.html
├── image-captioner.html
├── speech-transcriber.html
└── media-analyzer.html

Models and Download Sizes

Upon first execution, each model is downloaded and cached in the browser, allowing for offline operation thereafter. Here are the details of the models used:

Image Classification: The model used is Xenova/vit-base-patch16-224, which weighs approximately 88 MB.
Image Captioning: The model Xenova/vit-gpt2-image-captioning is used, with a size of about 246 MB.
Speech Transcription: The model Xenova/whisper-tiny.en weighs around 78 MB.

The combined application loads all three models, totaling about 400 MB on the first run. A progress indicator for each model is essential to provide a good user experience.

Task 1: Image Classification

Image classification involves assigning labels from a fixed set to an input image. The model used here is ViT-Base/16, a Vision Transformer trained by Google on ImageNet-21k and fine-tuned on ImageNet-1k, converted to ONNX format for use in the browser. It classifies images into 1,000 ImageNet categories and returns a ranked list with confidence scores.

Example Output

// Output from the classifier(imageUrl)
[
  { label: 'golden retriever', score: 0.9421 },
  { label: 'Labrador retriever', score: 0.0312 },
  { label: 'Sussex spaniel', score: 0.0098 },
  // ... top_k results in total
]

Each object has a label string (the name of the ImageNet class) and a floating-point score between 0 and 1. By default, the pipeline returns 5 results. Set top_k in the call to get more or fewer results.

Full Demonstration

Save this file as image-classifier.html in your project folder. Copy the code below and open it on your localhost.

<title>Image Classifier</title>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
  font-family: system-ui, sans-serif;
  max-width: 700px;
  margin: 2rem auto;
  padding: 0 1rem;
  background: #f8fafc;
  color: #1e293b;
}
h1 { margin-bottom: 0.25rem; font-size: 1.4rem; }
.subtitle { color: #64748b; font-size: 0.9rem; margin-bottom: 1.5rem; }
#status { font-size: 0.85rem; color: #64748b; margin-bottom: 1rem; }
.upload-area {
  border: 2px dashed #cbd5e1;
  border-radius: 8px;
  padding: 2rem;
  text-align: center;
  [cursor](/outil/cursor): pointer;
  background: white;
  transition: border-color 0.2s;
}
.upload-area:hover { border-color: #2563eb; }
.upload-area input { display: none; }
#preview {
  margin-top: 1rem;
  max-width: 100%;
  border-radius: 8px;
  display: none;
}
.result-row {
  display: flex;
  align-items: center;
  gap: 0.75rem;
  margin-top: 0.6rem;
}
.result-label { min-width: 200px; font-size: 0.9rem; }
.bar-bg {
  flex: 1;
  background: #e2e8f0;
  border-radius: 4px;
  height: 16px;
}
.bar-fill {
  background: #2563eb;
  height: 100%;
  border-radius: 4px;
  transition: width 0.4s ease;
}
.result-score {
  min-width: 48px;
  text-align: right;
  font-size: 0.85rem;
  color: #475569;
}
#results { margin-top: 1.25rem; }
#results h3 { font-size: 0.95rem; color: #374151; margin-bottom: 0.5rem; }
</style>
<h1>Image Classifier</h1>
<p class="subtitle">
  Upload any image -- ViT classifies it into ImageNet categories.
  Works entirely in your browser.
</p>
<div id="status">Downloading model (~88 MB on first launch)...</div>
<div class="upload-area" id="drop-zone">
  <p>Click to upload or drag an image here</p>
  <p style="font-size:0.8rem;color:#94a3b8;margin-top:0.4rem">
    JPG, PNG, WebP, GIF supported
  </p>
</div>
<img id="preview" alt="Preview" />
<div id="results"></div>
<script type="module">
import { pipeline } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.0.2';
const statusEl  = document.getElementById('status');
const dropZone  = document.getElementById('drop-zone');
const fileInput = document.getElementById('file-input');
const preview   = document.getElementById('preview');
const resultsEl = document.getElementById('results');

// ── Load the image classification pipeline ──────────────────────────
let classifier;
pipeline('image-classification', 'Xenova/vit-base-patch16-224', {
  dtype: 'q8',
  progress_callback: (p) => {
    if (p.status === 'progress') {
      statusEl.textContent = `Downloading model: ${Math.round(p.progress ?? 0)}%`;
    }
  }
}).then(pipe => {
  classifier = pipe;
  statusEl.textContent = 'Model ready. Upload an image to classify.';
  dropZone.style.borderColor = '#22c55e';
}).catch(err => {
  statusEl.textContent = `Error loading model: ${err.message}`;
});

// ── Classify an image from a data URL ───────────────────────
async function classifyImage(dataUrl) {
  statusEl.textContent = 'Classifying...';
  resultsEl.innerHTML  = '';
  try {
    const results = await classifier(dataUrl, { top_k: 5 });
    statusEl.textContent = 'Done.';
    let html = '<h3>Top Predictions</h3>';
    results.forEach(({ label, score }) => {
      const pct = (score * 100).toFixed(1);
      const bar = (score * 100).toFixed(0);
      html += `
        <div class="result-row">
          <span class="result-label">${label}</span>
          <div class="bar-bg">
            <div class="bar-fill" style="width:${bar}%"></div>
          </div>
          <span class="result-score">${pct}%</span>
        </div>`;
    });
    resultsEl.innerHTML = html;
  } catch (err) {
    statusEl.textContent = `Classification error: ${err.message}`;
  }
}

// ── File handling ────────────────────────────────────────────────────
function handleFile(file) {
  if (!file || !file.type.startsWith('image/')) return;
  const reader = new FileReader();
  reader.onload = (e) => {
    const dataUrl = e.target.result;
    preview.src     = dataUrl;
    preview.style.display = 'block';
    if (classifier) {
      classifyImage(dataUrl);
    } else {
      statusEl.textContent = 'Model loading -- please wait a moment and try again.';
    }
  };
  reader.readAsDataURL(file);
}

// Click to browse
dropZone.addEventListener('click', () => fileInput.click());
fileInput.addEventListener('change', (e) => handleFile(e.target.files[0]));

// Drag and drop
dropZone.addEventListener('dragover', (e) => {
  e.preventDefault();
  dropZone.style.borderColor = '#2563eb';
});
dropZone.addEventListener('dragleave', () => {
  dropZone.style.borderColor = '#cbd5e1';
});
dropZone.addEventListener('drop', (e) => {
  e.preventDefault();
  dropZone.style.borderColor = '#cbd5e1';
  handleFile(e.dataTransfer.files[0]);
});
</script>