X.ai, Elon Musk’s AI startup, has revealed its latest generative AI model, Grok-1.5. Set to power social network X’s Grok chatbot in the not-to-distant future (“in the coming days,” X.ai writes in a blog post), Grok-1.5 appears to be a measurable upgrade over its predecessor, Grok-1 — at least judging by the benchmark results and specs that X has published.
Grok-1.5 benefits from “improved reasoning,” according to X.ai, particularly where it concerns coding and math-related tasks. The model more than doubles Grok-1’s score on a popular mathematics benchmark, MATH, and scores over ten percentage points better on the HumanEval test of programming language generation and problem-solving abilities.
Of course, it’s difficult to predict how those results will translate in actual usage. As we recently wrote, commonly-used AI benchmarks, which measure things as esoteric as performance on graduate-level chemistry exam questions, do a poor job of capturing how the average person interacts with models today.
One improvement that should lead to observable gains is the amount of context Grok-1.5 can take in compared to Grok-1.
Grok-1.5 has a 128,000-token context — “tokens” referring to bits of raw text (e.g., the word “fantastic” split into “fan,” “tas” and “tic”). Context, or context window, refers to input data (in this case, text) that a model considers before generating output (more text). Models with small context windows tend to forget the content of even very recent conversations, while models with larger contexts avoid this pitfall — and, as an added benefit, better grasp the flow of data they take in.
“[Grok-1.5 can] utilize information from substantially longer documents,” X.ai writes in the aforementioned blog post. “Furthermore, the model can handle longer and more complex prompts while still maintaining its instruction-following capability as its context window expands.”
What’s historically set X.ai’s Grok models apart from other generative AI models is that they respond to questions about topics that are typically off-limits to other models, like conspiracies and more controversial political ideas. The models also answer questions with “a rebellious streak,” as Musk has described it, and outright rude language if requested to do so.
It’s unclear what changes, if any, Grok-1.5 brings in these areas. X.ai doesn’t allude to this in the blog post.
Grok-1.5 will soon be available to early testers on X, X.ai says, accompanied by “several new features.” Musk has previously hinted at summarizing threads and replies and suggesting content for posts; we’ll see if those arrive soon enough.
The announcement of Grok-1.5 comes after X.ai open sourced Grok-1, albeit without the code necessary to fine-tune or further train it. More recently, Musk said that more users on X — specifically those paying for X’s $8-per-month Premium plan — would gain access to Grok, the chatbot, which was previously only available to X Premium+ customers (who pay $16 per month).