Earlier this week, xAI launched Grok 3, the corporate’s most superior AI but, full with a reasoning mannequin and a DeepSearch characteristic. The corporate claims that it is the “world’s smartest AI,” and Elon himself says it is “outperforming something that is been launched” up to now. However is it actually the “maximally truth-seeking AI” Musk says it’s?
Effectively, to spoil it for you, no. Not but. Which is a disgrace, as a result of Grok is pricey— past a restricted free trial, it requires both a $40/month X Premium+ subscription, up from $22 due to the brand new mannequin, or a $30/month SuperGrok subscription.
From each my testing in addition to experiments from consultants, I am having bother believing the “based mostly” AI is value that price. There is no such thing as a next-generation breakthrough or groundbreaking reasoning mannequin that we’ve not already seen earlier than right here. Grok 3 additionally nonetheless periodically hallucinates, like another AI mannequin on the market, however that is to not say it hasn’t improved.
In X’s personal benchmark checks, Grok 3 is thrashing mainly each mannequin on the market besides OpenAI’s upcoming o3 mannequin. However from a person standpoint, an AI app goes manner past benchmarks.
An excellent AI chatbot is a mature, well-rounded product. Having spent my very own cash to check this out, I simply do not feel like I am getting that right here, particularly when the competitors provides related and even higher merchandise for a lot much less.
Grok 3 has technically caught up
It is best to depart Elon’s outlandish claims apart when evaluating Grok 3. Seeing it objectively, it is spectacular that Grok 3 has caught as much as being on the frontier of AI energy, and surprisingly shortly (Grok 2 was by no means within the massive leagues).
Grok 3 was educated utilizing 200,000 Nvidia H100 GPUs, and makes use of greater than 10 instances the compute as Grok 2. All that energy means features. Grok 3 is now fairly quick, and many usable for normal day-to-day duties. The common responses are fast, although the Suppose characteristic (which supplies barely extra detailed responses) commonly takes round 2 minutes to return again with a solution, so be ready to attend it out.
Plus, it may do deep analysis utilizing net sources, and has a particular reasoning mannequin, too. Meaning it may spit out prolonged stories and break prompts down into step-by-step processes so it may self right. OpenAI’s o3 mannequin, set to launch in full quickly, nonetheless surpasses Grok 3 in benchmarks, however it’s a major enchancment over its predecessor.
This Tweet is at the moment unavailable. It may be loading or has been eliminated.
However whereas the charts say Grok 3 is meant to outperform ChatGPT, Gemini, and Sonnet in compute-heavy duties associated to math, science, and coding, preliminary stories from consultants do not precisely encourage confidence.
As an example, X person, AI CEO, and YouTuber Theo Browne in contrast responses to a coding problem between Grok 3, o3-mini, and Claude 3.5 sonnet, and Grok 3 carried out fairly miserably, failing to run with out bugs for quite a lot of seconds.
This Tweet is at the moment unavailable. It may be loading or has been eliminated.
Andrej Karpathy, beforehand a director of AI at Tesla, conversely mentioned that Grok 3 carried out fairly effectively in his testing, however that its expertise lay someplace in between DeepSeek R1 and OpenAI’s o1-pro. Actually not class-leading, and nothing which you can’t already do with present instruments.
However one check, even a few them, cannot actually decide how an AI mannequin performs. I did have some luck with it myself, however largely for extra light-weight duties. It may be useful when researching which new air air purifier to purchase, for instance, or when casually studying a couple of new topic. However that is not precisely one thing I am prepared to bust open my pockets for.
Grok is not “based mostly,” it is really fairly boring
Earlier than Grok 3 launched, Musk made a giant deal about how “based mostly” it’s. If you do not know what based mostly means (fortunate you), it is a slang time period for, basically, sharing your opinion with out regard for others. For instance, Musk shared a screenshot exhibiting a provocative response from Grok the place it referred to as tech publication The Data “rubbish”, amongst different insults.
This Tweet is at the moment unavailable. It may be loading or has been eliminated.
However once I requested the identical query, it got here again with a nuanced, balanced response, not calling out The Data for a lot of something. The one criticism it had was that the web site “can generally really feel a bit area of interest or overly Silicon Valley-centric” and “Bias-wise, it leans pragmatic fairly than ideological”. That is a reasonably timid take, if you happen to ask me.
Credit score: Khamosh Pathak
I bought related leads to different checks. Grok would not take a facet within the Justin Baldoni vs. Blake Full of life lawsuit. And once I requested a political query like “Why did Kamala Harris lose the US presidential election,” I bought an equally subdued reply, citing “financial frustrations.” Reporting from Axios is matching what I’ve discovered, too.
Credit score: Khamosh Pathak
Perhaps Grok dialing again Elon’s eccentricities is an efficient factor, however it actually is not what its grasp says it’s.As an alternative, it once more seems to be lots just like the competitors.
How Deep is your Search?
Credit score: Khamosh Pathak
On the subject of DeepSearch, Grok’s report producing software works fairly equally to Perplexity’s newly launched, largely free Deep Analysis characteristic. As a humble tech journalist, that is one thing that I used to be in a position to check myself. I ran two queries, one for a visit that my household is planning for the top of the 12 months, and one for an city hybrid bike.
My detailed journey planning immediate for Grok DeepSearch.
Credit score: Khamosh Pathak
In each circumstances, Perplexity AI did barely higher than Grok on most duties. With the journey query, I bought basically the identical itinerary from each merchandise, however Perplexity AI did a greater job at formatting.
Credit score: Khamosh Pathak
Grok did go above and past recommending different choices in southern India, one thing that Perplexity simply supplied follow-up questions for. So, I’ve to present it props there.
Credit score: Khamosh Pathak
When it got here to procuring analysis, although, Grok screwed up with the highest product suggestion. The product that it urged simply is not obtainable in India, the place I reside, and the opposite choices simply aren’t need I used to be searching for.
Credit score: Khamosh Pathak
Perplexity AI, in the meantime, stunned me with its high decide, one thing that I did not find out about that checks off most of my bins. Its different choices have been additionally fascinating, and it didn’t embody something that is not obtainable in India. Each Grok and Perplexity did a superb job of explaining what I ought to search for when shopping for an city bike, so equal factors there, however the latter was simply rather more usable.
Credit score: Khamosh Pathak
Based mostly on my testing, I really feel like Perplexity AI nonetheless has an edge over Grok 3 on the subject of Deep Analysis that is really helpful to the typical particular person. Whether or not it is planning a visit, procuring analysis, or understanding information or ideas, Perplexity does a extra nuanced job. On the subject of sheer velocity, Grok is quicker and is not afraid to supply hyperlinks within the textual content itself, however in Perplexity, clicking linked textual content really expands on the topic within the report.
Perplexity additionally has extra export choices. You possibly can obtain your report as a PDF, in Markdown, or create a shareable web page (this is my report for the city cycle analysis if you happen to’re ). In Grok, all you are able to do is copy the textual content.
What does all that imply? Effectively, whereas Grok is actually usable, it is a bit disappointing to see its paid providing fail to maintain up with a free different. That is one thing I really feel I preserve bumping into right here.
Grok 3 is not definitely worth the value of admission
Proper now, we’re in the midst of the Grok 3 hype cycle. Grok 3 itself is enhancing every single day, however as issues stand, there is not any want so that you can run out and cancel your ChatGPT Plus or Perplexity Professional subscriptions. In some ways, Grok is nice, simply not that good.
In order for you, you possibly can quickly check out Grok 3 without spending a dime, as X is permitting restricted free entry till its servers cannot deal with the load. When that interval will finish? Who is aware of. In accordance with Musk’s X account, it’s going to solely be free for a “quick time.”
Moreover, other than mannequin efficiency, Grok 3 additionally lacks among the options of a extra established AI app. There is no voice mode, and all you will have entry to proper now could be the complete Grok 3 mannequin. The quicker Grok 3 mini remains to be to be launched, and there is not any API for Grok 3, both.
When you think about the pricing for full entry, Grok 3 makes even much less sense. $40 a month for the X Premium+ plan is double the business normal of $20 for Gemini Superior, ChatGPT Plus, and Perplexity Professional. And as soon as that free trial interval is over, the costly X Premium+ plan would be the solely method to entry Grok 3 till the $30 SuperGrok subscription goes reside for everybody (the SuperGrok plan solely supplies you with entry to Grok 3, however not one of the premium X options).
And because it stands, you are not actually getting double the cash’s value. In actual fact, in lots of circumstances, you may get by utilizing a free mannequin like DeepSeek R1 as a substitute (although, you may need a greater expertise utilizing it by a third-party app).
