NVIDIA Releases Garak to Safeguard LLMs

5 months ago 49

Last updated November 18, 2024
In AI News

It evaluates LLMs for vulnerabilities such as hallucinations, data leaks, prompt injections, misinformation, toxicity, and jailbreak scenarios.

NVIDIA has launched Garak, an open-source vulnerability scanner designed to identify potential weaknesses in LLMs. Dubbed the “nmap for LLMs,” Garak acts as a red-teaming and assessment tool for generative AI systems.

Click here to check out the GitHub repository.

Possibly named after Elim Garak from Star Trek, who is an exiled spy from the Cardassian Union, NVIDIA’s Garak also performs similar tasks like the fictional character. It evaluates LLMs for vulnerabilities such as hallucinations, data leaks, prompt injections, misinformation, toxicity, and jailbreak scenarios.

Garak employs static, dynamic, and adaptive probing techniques to simulate failure modes in AI models and dialogue systems. The tool is free to use, and NVIDIA is actively enhancing its features to support a wider range of applications.

Currently, Garak supports Hugging Face Hub generative models, replicate text models, and OpenAI API chat and continuation models. It also supports anything accessible via the REST API.

“While most of the recent LLMs, especially commercial ones, are aligned to be safer to use, you should bear in mind that any LLM-powered application is prone to a wide range of attacks,” said NVIDIA in the release post.

Last month, NVIDIA launched the Nemotron-4-Mini-Hindi-4B model, a small language model for Hindi, enabling businesses to deploy AI solutions specific to local needs. This model, part of NVIDIA’s NIM microservice, can be deployed on NVIDIA GPU-accelerated systems, optimising performance for various applications.

The chip giant also introduced HOVER (humanoid versatile controller), a 1.5 million parameter neural network designed to coordinate the motors of humanoid robots for locomotion and manipulation.

(Total 1 views)

Mohit Pandey

Mohit writes about AI in simple, explainable, and sometimes funny words. He holds keen interest in discussing AI with people building it for India, and for Bharat, while also talking a little bit about AGI.

Association of Data Scientists

GenAI Corporate Training Programs

India's Biggest Developers Summit

February 5 – 7, 2025 | Nimhans Convention Center, Bangalore

Download the easiest way to
stay informed

Subscribe to The Belamy: Our Weekly Newsletter

Biggest AI stories, delivered to your inbox every week.

Rising 2025 | DE&I in Tech & AI Summit

Mar 20 and 21, 2025 | 📍 J N Tata Auditorium, Bengaluru

Data Engineering Summit 2024

May 30 and 31, 2024 | 📍 Bangalore, India

February 5 – 7, 2025 | Nimhans Convention Center, Bangalore

MachineCon GCC Summit 2024

June 28 2024 | 📍Bangalore, India

September 25-27, 2024 | 📍Bangalore, India

25 July 2025 | 583 Park Avenue, New York

Our Discord Community for AI Ecosystem, In collaboration with NVIDIA.

World's Biggest Media & Analyst firm specializing in AI

AIM publishes every day, and we believe in quality over quantity, honesty over spin. We offer a wide variety of branding and targeting options to make it easy for you to propagate your brand.

AIM Brand Solutions, a marketing division within AIM, specializes in creating diverse content such as documentaries, public artworks, podcasts, videos, articles, and more to effectively tell compelling stories.

ADaSci Corporate training program on Generative AI provides a unique opportunity to empower, retain and advance your talent

With MachineHack you can not only find qualified developers with hiring challenges but can also engage the developer community and your internal workforce by hosting hackathons.

Conduct Customized Online Assessments on our Powerful Cloud-based Platform, Secured with Best-in-class Proctoring

AIM Research produces a series of annual reports on AI & Data Science covering every aspect of the industry. Request Customised Reports & AIM Surveys for a study on topics of your interest.

Immerse yourself in AI and business conferences tailored to your role, designed to elevate your performance and empower you to accomplish your organization’s vital objectives.