Plan for Losing It: An AI Vendor Risk Assessment in Four Questions

Darth Vader in an executive boardroom saying 'I'm altering the deal. Pray I don't alter it any further.' to humorously illustrate the nee for AI vendor risk assessment

Most AI vendor risk assessment processes check security certifications and data handling. Yours should also check what happens to your business if the vendor stops supplying the model.

Not long ago, we assisted a company seeking to move their website to a trending AI platform. Everyone was using it and it had recently been purchased by a major tech company that said it was planning to integrate it into all their offerings. A couple of months later, the deal was blocked by the foreign owner of the AI platform and everyone in the world who had bet the farm on it scrambled to move to something else.

Fortunately for our client, we had advised them not to put all their eggs in that particular basket. We were able to quickly pivot to a custom solution with a more reliable AI vendor instead. And we built their solution so that if that vendor also disappeared or changed the terms of service, we could still maintain and operate it (albeit somewhat less efficiently than AI) until we found a replacement.

Below is what happened to Cursor’s customers who hadn’t done a thorough AI vendor risk assessment, and how to avoid the same fate.

Two vendors argued, and their customers paid

On August 29, 2026, OpenAI told Cursor it would stop supplying its models on November 12.

SpaceX had just bought Cursor’s parent company, and OpenAI said it did not expect the new owner to follow its terms of service. That was the whole explanation. Cursor’s customers had signed nothing with OpenAI and had no say in the acquisition, and they got about ten weeks to move their work to other models or to another platform.

The analysts InfoWorld spoke with expected a productivity dip during the move. Prompts you tuned for one model give inconsistent results on another, and you have to retest every agent configuration and tool integration against the replacement. Pareekh Jain of Pareekh Consulting said enterprises can no longer assume that a model available through an AI coding platform today will always remain available.

If your company runs a support assistant, a quoting tool, a document pipeline or an internal search box on one vendor’s model, you were in the same position on August 29 without knowing it. You signed for the feature, and you cannot say how many weeks a forced switch would take. When the vendor’s letter arrives, their notice period sets your schedule.

GitHub switched Copilot to metered billing two months later

On October 28, GitHub announced that Copilot now comes with every GitHub Enterprise license, and that it meters AI usage from the first request against an enterprise budget. There is no included allowance. Code completion keeps working when the budget runs out; chat, code review and cloud agents stop until you add money. If your team doesn’t have an Enterprise license, you can buy Copilot Standalone at $10 per user per month, with no monthly AI credits attached.

GitHub announced the change and put it into effect on the same day. You used to pay a flat subscription, and now you pay by the request.

If you approve that budget, you now need to know which of your workflows burn through it, and you had no reason to ask that under the flat subscription.

Stack Overflow’s 2026 Developer Survey found that 66 percent of developers use coding assistants or agents. A third kept the same tool over the past year, a quarter switched for better output, and one in ten switched because their organization required or recommended it. Switching tools has become routine, and the companies that handled it well had planned for it before vendors announced anything.

A four-question AI vendor risk assessment

You do not need to read code to answer the questions below. Ask the vendor, the agency, or the employee who built the feature, and write down what they say.

  1. Where do the prompts live? The instructions that shape every answer the model gives are the most valuable part of the build. If a developer scattered them through the application code, you pay a developer to hunt through every file when the model changes. If you keep them in one place under version control, with a note on what each one is for, you can edit and retest them without a project plan.
  2. Is there an evaluation set? A file of real inputs with outputs a person has already judged acceptable, which you can run against any model. Without one, you find out the replacement model is worse when a customer complains.
  3. How is the model called? Through one layer that reads the model’s name and the vendor’s address from a settings file, or hard-wired in every place the application needs an answer. With the settings file, you change one line and rerun the evaluation set.
  4. Where does the data go, and can it come back? Conversation logs, uploaded documents, fine-tuning data and usage history live on the vendor’s side. Ask what you can export, in what format, at what cost, and how long after cancellation it is still available.

Your answers give you your switching cost in weeks. If the person you ask has to go and look, add a week to the estimate.

The evaluation set costs the least and protects the most

Most companies answer no when I ask about the evaluation set, and it is the cheapest part to fix.

Build it from your own records: customer emails paired with the replies you would be happy to send, or contracts paired with the clauses your lawyer flagged. The AI feature runs the inputs, and a person compares what comes out against the file. From then on, you know whether the feature works today, which you have been taking on faith since launch, and you have a figure to show the vendor when quality drops after a model update.

When a migration comes, the candidate model runs the same file, and you see the difference before customers do.

Your domain expert, the person who knows what a good answer looks like in your business, builds the set in a few days with no engineering time. You end up owning the definition of good, and the model becomes one of several that can meet it.

Put the model’s name in a settings file and call it from one place

You have to decide on the second cheapest protection when you first build the feature. Retrofitting it later costs several times as much.

Every call to the model goes through one layer, and that layer reads the model’s name and the vendor’s address from configuration. When the vendor changes, you change the configuration, rerun the evaluation set, and the rest of the application won’t be affected. The same layer counts usage, so after a move to metered billing like GitHub’s, you can see which workflow draws down the budget and put a ceiling on it. If your calls are scattered across scripts, spreadsheets, plugins, and browser extensions, you get your first clear view of spending from the invoice.

When we scope an AI build for a client, this layer is in the first estimate and adds days to the schedule. Retrofitting it into an application that an employee vibe-coded on their own, or that a vendor assembled with no reason to make leaving easy, is the expensive version of the same work. Ask any builder, including a consultancy, where this layer is in their proposal, since a proposal without it covers the demo and leaves the next several years to you.

Four things to get in writing for an AI risk assessment

Cursor’s customers had no contract with OpenAI, so the notice period that decided their next ten weeks was in a document they had never seen. An AI vendor risk assessment that neglects contract reviews misses that clause. Before you sign with an AI vendor, or with a builder who will put a vendor’s model inside your product, get these in writing.

  • Notice for model retirement or access changes. Ninety days is a reasonable ask. Vendors retire model versions on their own schedule, and a published deprecation policy with dates beats a verbal promise.
  • Who sits underneath. A platform that resells another company’s model should name it and say what happens to your service if that relationship ends.
  • Data export on exit. Format, timing, and cost, with logs, documents, fine-tuning data, and evaluation results coming back in a form another vendor can load.
  • Notice on price or metering changes. GitHub announced its change and put it into effect the same day. Ask for thirty days, and for usage reports, you can read without the vendor’s dashboard.

Ten weeks turned out to be the number in the Cursor case. With the evaluation set and the configuration layer in place, you can test a replacement and go live inside that window. Skip both, and you spend those weeks on a rebuild under deadline, with the rebuilder setting the price.

Run the AI vendor risk assessment in one meeting

List every AI feature your company depends on, from the chatbot on the website to the script that drafts invoices, and rate each one against the four questions. An hour or two with the people who know the answers is enough for most companies.

QuestionLow exposureMedium exposureHigh exposure
Where do the prompts live?One place, under version control, documentedOne place, undocumentedScattered through code or in someone’s head
Is there an evaluation set?Yes, run on every changeExists, run rarelyNone
How is the model called?One layer, model named in configurationOne layer, model named in codeDirect calls from many places
Where does the data go?Export tested, format knownExport promised in contractUnknown

A feature with two or more answers in the High column needs work before the next vendor letter, and the evaluation set comes first because it makes every other fix measurable. Once every answer is in the Low column, you can swap models on that feature as routine maintenance, and that status belongs in your risk register next to the feature’s name.

If you run an agency or consultancy, score your own delivery standard with the same table. When builds you shipped in the past two years would score High, the migration work arrives in your inbox the next time a vendor changes terms, at emergency rates, and your client blames you.

Plan for the switch before the build

Every AI feature a business depends on will change models at some point, through a vendor dispute, a price change, a retirement notice or a better model arriving. The companies that come through more quickly have an evaluation set, a configuration layer, a tested data export and a contract with notice periods in it. All four cost less up front when scoping the feature than on the day the notice arrives.

When an AI feature is already in production at your B2B or B2C company, we run this AI vendor risk assessment against it, build the evaluation set with your domain expert, and give you a switching plan with the week count on it. For a build you are about to commission, we scope the configuration layer and the evaluation set into the first estimate and put the contract questions to the vendor before you sign. Agencies get the same review set up as a delivery standard, and we work alongside your team on it.

Send me a message about the feature, and I will tell you within one business day what the review would cover.

Sources

More Articles

Leave a Reply

Your email address will not be published. Required fields are marked *