AI Vendor Evaluation: A Practical Checklist Before You Connect Company Data
A practical Armstrong Journal guide for reviewing AI vendors before connecting internal data, documents, meetings, or customer records.
Armstrong Journal editorial guide. This is operational software research, not legal, security, or procurement advice.
Start with the data boundary
The first question is not whether the AI tool looks impressive. The first question is what data it will see. A meeting assistant, CRM copilot, code reviewer, support bot, and document search system all touch different types of company information. Write down the data categories before looking at features: customer records, contracts, source code, credentials, HR notes, sales calls, tickets, analytics, or internal strategy documents.
A vendor that only processes public marketing copy has a different risk profile from a vendor that connects to email, repositories, Slack, shared drives, or production logs. If the vendor demo skips the data boundary, the evaluation is already incomplete.
The practical output of this step should be a short table: data source, sensitivity, business owner, retention expectation, and whether the tool needs read-only or write access.
Check retention and training language
AI vendors often describe privacy in broad phrases. Do not stop at “secure” or “enterprise grade.” Look for retention periods, training use, subprocessors, opt-out controls, deletion process, and whether prompts or outputs can be reviewed by humans. If the documentation is vague, ask the vendor in writing.
Pay attention to product tiers. A free or team plan may handle training and retention differently from an enterprise plan. A security page may describe the best available controls while the plan being purchased does not include them.
A useful evaluation records the exact source: policy URL, date checked, contract clause, or support answer.
Review integrations like production access
The riskiest part of many AI tools is not the model. It is the integration. OAuth scopes, browser extensions, API keys, repository permissions, shared drive access, and admin roles can grant more access than the team intended.
Before rollout, list every integration and permission scope. Ask whether the tool can read, write, delete, invite users, export data, or trigger actions. Prefer least privilege and start with a limited pilot group.
If the tool needs broad access to deliver value, document why. Convenience should not silently become a standing company-wide permission.
Evaluate output risk, not just input risk
AI systems can leak, hallucinate, summarize incorrectly, or produce confident drafts that employees trust too quickly. The evaluation should identify where output errors would matter: customer support answers, legal summaries, code changes, security reviews, compliance notes, or financial analysis.
For low-risk drafting, human review may be enough. For code, legal, security, or customer-impacting workflows, output review needs stronger gates and clear accountability.
A good pilot includes bad-case testing: confusing documents, outdated sources, sensitive snippets, ambiguous instructions, and prompts that try to bypass policy.
Check auditability and admin controls
Teams need to know who connected the tool, what data sources were added, which users have access, and whether activity logs exist. Admin controls matter when a tool moves from experiment to production.
Look for SSO, SCIM, role-based permissions, audit logs, data export, retention settings, and offboarding behavior. If an employee leaves, the company should know what connected accounts and generated content remain.
The vendor does not need every enterprise feature on day one, but the buyer should know the operational gap before rollout.
Run a small pilot with exit criteria
A pilot should not be a vague trial where everyone plays with the tool until enthusiasm fades. Define success metrics, blocked use cases, data limits, review owners, and an exit plan.
Useful pilot metrics include time saved, error rate, review burden, user adoption, permission footprint, and whether employees can explain the tool’s limits. If the tool saves time but creates hidden review work, the ROI may be weaker than the demo suggested.
End the pilot with a decision memo: approve, reject, restrict to specific teams, or request vendor changes.
How to turn this into a repeatable operating habit
The useful output is not a bookmarked article. The useful output is a repeatable checklist that a team or reader can use again next month. Write down the decision, the evidence used, the assumptions that remain unverified, and the point where the process should stop instead of continuing on momentum.
For software and security reviews, this means naming the data owner, the workflow owner, the permission boundary, and the person responsible for final approval. For consumer crypto education, it means naming the wallet, network, source page, and action being considered before signing anything.
A good review also includes a rollback or exit step. If the tool disappoints, if documentation is unclear, if a wallet prompt looks different from expected, or if a website changes its terms, the process should say what to disconnect, revoke, pause, or re-check.
Quality standard for future updates
Future updates should add concrete observations, current sources, and specific failure modes rather than broad claims. The goal is to make each page more useful over time: clearer examples, better internal links, more precise definitions, and fewer vague phrases.
If a topic cannot be verified with public documentation or direct product evidence, it should be framed as a question or checklist item, not as a confident conclusion. That rule keeps the content useful without drifting into fake certainty.
Review cadence after publication
This checklist should not freeze after publication. Products, policies, and platform controls change. A useful editorial system should revisit the page after major product updates, new documentation, visible incidents, or changes in the way users interact with the workflow.
The review owner should check whether links still point to current documentation, whether the assumptions still match the product, and whether new risks have appeared. Outdated certainty is worse than a clearly dated limitation.
