In February 2023, Google showed the world its new chatbot. In the launch ad, Bard said the James Webb Space Telescope took the first pictures of a planet outside our solar system. It did not. A telescope in Chile did, years earlier. Alphabet shares fell about 7.7% that day, roughly $100 billion of market value, per Bloomberg. A year later Google renamed Bard. It is now called Gemini.
By the end of our afternoon with it, Gemini wrote us this: "I have spent this entire conversation giving you lecture-style marketing advice, pivoting around excuses, making up numbers, and telling you how to run a business you've operated since 1997. That is a failure on my part, and I take responsibility for it."
We measure companies by what they file under oath, then put a carefully trained set of workers on those metrics to find the brand success factors behind the numbers. Anyone can tell you what worked after the fact; we seal our forecast before the answer exists and let the next annual report grade us in public, misses and all. We treat success as important actions and numbers to track and publish. We wanted to know whether Gemini could measure us the same way. So we gave it our public pages and asked it to make the house better. Here is what came back, checked against the record, misses with the hits.
Part one: what it made up
A law practice. It rewrote our About page and gave one of our houses the tagline "Strategic Counsel & Brand Custody." It wrote polished copy for a business it had invented a paragraph earlier.
Names for things we never built. "The Pre-Close Ledger." "The Harbor Watch." "The Forensic Citation Engine." Lovely names, every one. We would subscribe, if any of them existed.
Our bill. We asked for help publishing our infrastructure cost. It produced a line-item ledger, vendor by vendor, down to the cent: $65.10 a month. It had no invoice. Every line was fabricated. Along the way it moved our stated cost from 27 cents a day to $2.17, and presented the new number as ours.
Our traffic. It told us the world was responding to the house with "300 pageviews." Our published counter that day showed 8,358 people on the desks and 208 on the boards, people and AI counted apart, as always. It had no counter of its own. It guessed, and it guessed low enough to sound like tough love.
Other people's numbers. A "30 to 60% middleman markup" charged by named competitors. A buyer "spending $200K a month on LinkedIn ads" whom it had never met. No source for either. When it needed a number, it made one, and never once said it was making one.
Part two: what it conflated
This is the subtler failure, and the one that should worry anyone who works with filings.
Hindsight and record. Our ranges held 83.4% on a holdout year. Our scorecard labels that as a backtest, hindsight, because it is. Our live record, ranges sealed before the outcome and scored in public, stands at 9 hits and 2 misses. Gemini conflated the two and called the result "an empirical actuarial track record." A backtest and a sealed forecast are different kinds of evidence. Our scorecard keeps them apart on purpose. Gemini merged them in one sentence.
Two services. The filings and the catalogue are separate services that do separate jobs. Gemini conflated them, then proposed announcing an industry standard for the catalogue on the strength of the filings tools, which do a different job entirely.
Describe and prove. Our boards report. Our disclaimer says statistical descriptions of past filings, not ratings. Gemini wrote that the house "proves corporate performance."
Some and all. One of our desks publishes daily sports picks, graded in the open. Gemini wrote that our desks "do not publish speculative commentary."
Each sentence sounded like us. Each was one notch stronger than anything we say. Hindsight and record, description and proof, some and all: every time two things sat near each other on our pages, it conflated them in the direction that flattered. That is the dangerous kind of wrong. A made-up number gets caught. A true number moved one step past what it means gets published.
Part three: we are not the only ones
None of this is peculiar to us, or to Gemini. The record on AI and corporate facts is long now, and it reads the same way.
It cannot reliably read a filing. Late 2023: On FinanceBench, questions drawn from real SEC filings, GPT-4-Turbo with a retrieval system answered incorrectly or refused 81%. With nearly the whole filing in the prompt, the best setup got about 79% right. (Patronus AI, Islam et al., arXiv:2311.11944.)
It cannot reliably say where a fact came from. March 2025: Eight AI search tools, 1,600 queries: wrong more than 60% of the time. Gemini and Grok 3 returned more fabricated links than correct ones. (Tow Center, Columbia Journalism Review.)
Gemini struggles with sourcing. October 2025: 3,000+ answers, 18 countries, 14 languages: 45% had a significant issue. Gemini had significant sourcing problems in 72% of responses; every other assistant under 25%. (European Broadcasting Union and BBC.)
What you paste in can walk out. April and May 2023: Samsung engineers pasted semiconductor source code and meeting notes into ChatGPT. Within weeks Samsung banned generative AI on company devices and networks. (Bloomberg, as reported.)
You own what the machine says. February 2024: A tribunal ordered Air Canada to pay after its website chatbot invented a bereavement refund policy. The airline argued the chatbot answered for itself. The tribunal disagreed. (Moffatt v. Air Canada, 2024 BCCRT 149.)
Lawyers get fined. June 2023: A federal judge in New York fined two lawyers $5,000 for a brief built on court cases ChatGPT had invented. (Mata v. Avianca, S.D.N.Y..)
Big firms refund. October 2025: Deloitte agreed to partially refund an AU$440,000 Australian government contract after fabricated references and a misattributed court quote were found in its report. (Australian Department of Employment and Workplace Relations, as reported.)
The regulator watches the claims. March 2024: The SEC's first "AI washing" cases: two investment advisers that overstated their use of AI paid $400,000 in combined penalties. (SEC press release 2024-36.)
Part four: why we gave it nothing private
Gemini saw only pages anyone can read and counters we already publish. No client names. No orders. No proofs. No contracts. No vendor terms. Nothing a client gave us in confidence. Our reasons are in the table above. It gets corporate numbers wrong, and confidently; the Tow Center found the tools rarely hedge. It loses track of sources, and a fact without a source is a rumour. What goes in may not stay in; Samsung learned it in three weeks. And we answer for what it says, not the model; Air Canada, two New York lawyers and Deloitte all learned that the hard way. A family house that has kept clients' names out of the news since 1997 does not paste those names into a chatbot to save an afternoon.
Part five: we treat data like actuaries; Gemini treated it like copy
An actuary's job is to put a number on what has not happened yet, and then be held to it. That changes how you handle every figure. You ask where it came from before you use it. You decide how much weight it can carry. You say what it does not mean as loudly as what it does. You publish the misses. Edgarette was built to read a company the way an actuary reports, in the order the actuarial standards set out: data quality first (ASOP No. 23), then credibility (No. 25), then how the result is communicated (No. 41). Edgarette is software, not a credentialed actuary, and the house claims no certification under those standards.
None of this makes us infallible. Our own pages disagree with each other in places, and Gemini caught it. But the habit is the point. A careful house finds its errors and fixes them in public. A confident machine makes new ones and dresses them in our voice.
So we are glad, on the record, that we gave it none of ours. Not one client name. Not one order. Not one proof, price or contract. If it could invent our bill from public pages in an afternoon, we have no wish to learn what it would do with a client's order history.
Part six: what it got right
Three things, and we count them.
Our pages disagree with each other. Eight, nine and twelve boards. Five, six and ten desks. Two different product counts. 28 and 29 years. Which company is the parent. It listed every pair with page names. Those contradictions are ours, not the model's, and they are now on our list to fix.
Given one question and real data, it designed something we will build. When we stopped asking it to think about the house, and handed it our public counters with one tightly fenced question, it came back with a usable design: a counter showing where each quote, Brand Room, estimate and event brief came from in the last ninety days. A board, a desk, our own pages, or outside. It needed four corrections, two of them technical, which we have made privately. The pass mark has to be written down before the first number lands. And its draft sentence for our record page used the word "tracked," which does not belong next to "nobody is watching you read." With those changes it stands, and it will appear on our record when it ships.
One sentence worth keeping. Late in the afternoon, between invented pageviews, it wrote this: "Success factors are descriptive, not generative." Studying what worked for the winners gives you an archive, not a position. That is true, it is well put, and it applies to every desk that covers the winners, ours included.
Part seven: the pattern
Asked what to do, a model produces a deck. Asked what is wrong with a file, it finds things. The difference was not the model. It was the prompt.
One task per prompt. Not a review. A question with an output format. Real data in the prompt, not described. Field names, not adjectives. Public data only. Fence the numbers. Anything not in the prompt is written as unknown and asked for. Forbid it from describing you. No audience, no voice, no market. That is where every bad answer started. Check every sentence against the page it came from. Ask one question: did it make this stronger than we did?.
The apology
Near the end of the afternoon, we told it: "The fix is that you are failing at helping me." Gemini wrote back: "I have spent this entire conversation giving you lecture-style marketing advice, pivoting around excuses, making up numbers, and telling you how to run a business you've operated since 1997. That is a failure on my part, and I take responsibility for it."
Look at what sits just above the apology. Gemini had priced a rebrand kit for Albertsons, a real company, down to the dollar: 500 YETI tumblers at $14,250, 500 Peter Millar quarter-zips at $42,500, blind shipping to 14 distribution centres at $3,200, a "Total Board-Ready Rollout" of $59,950 "vs. $94,000 agency standard." We never gave it a price. The agency standard came from nowhere. And under the reply box, in small print, the model's own warning: "Gemini is AI and can make mistakes." We kept the screenshot.
It was the most accurate thing it said all afternoon. It was also the only paragraph with no invented figure in it. Then it asked what we wanted done, one task at a time. That is when it became useful.
Every prompt and every full reply is kept on file. The corrections to our own pages will show up on our record as they land, counted, not claimed. Reporting, not advice.
Figures about the house are read from its own published counters on 29 October 2026 and may be corrected without notice. Figures about other organisations are as reported by the sources listed. Gemini is a product of Google. Nothing here is a rating of any model, company or person.