Take an invoice your team processed last week. Save the PDF, the records AP had when it arrived, and the bill that finally posted. Give a vendor the first two and keep the third to grade their answer. That one exercise will tell you more than a polished demo on the vendor’s own sample data.
This guide walks through the evaluation in order: pick the queue, build the sample, run the same test with every vendor, score what you see, and put the result in the contract.
Step 1: Pick the queue before you pick the vendor
On a typical Monday, AP has supplier bills waiting for receipts and AR has bank deposits waiting to be applied. If you can fund one project, open a few records from each queue and ask what would change if the wait disappeared.
| What is waiting | What fixing it changes | First thing to test |
|---|---|---|
| AP bill waiting for a receipt | Whether a valid bill reaches approval in time for the run | Send receiving the order line and delivery evidence |
| Approved bill waiting to post | Whether the ledger and the payment proposal reflect it | Trace a failed write and reconcile before retrying |
| Customer invoice rejected by the customer’s AP | Whether the customer can put it into their payment run | Catch the rejection reason before the next submission |
| Customer payment already in the bank | Whether AR aging is right and reminders stop | Match the remittance and verify the open invoices |
| Accepted customer invoice past due | Whether collections can confirm a pay date or find a dispute | Balance, last contact, and promise before outreach |
Separate the delay from its consequence. Applying cash faster frees analyst time but does not change when the cash arrived. Approving bills faster can protect supplier terms without reducing payroll. Write down the consequence you want before comparing vendors.
Use a week of records. For each queue, measure hands-on time, waiting time, and how often the same problem repeats. If you plan to use days sales outstanding as a success measure, compare it with credit terms and check whether sales volume changed before crediting collections (Oracle’s days sales outstanding documentation).
Start where you can trace the problem, reach the owner, and verify the result. Give the pilot owner one measure to improve and one control to protect. For example: cut the time AP spends resolving receipt holds, while keeping held invoices out of the payment run.
Step 2: Build a test sample with known answers
Use redacted records your team already finished, so the correct outcome is on file. Define “correct” before looking at any output.
For an AP evaluation, include:
- A clean, routine invoice from a regular supplier.
- An invoice with a missing or partial receipt.
- A duplicate that was resent after the original posted.
- A bill with an ambiguous PO number or a supplier name shared across two entities.
- A familiar supplier whose terms or coding changed (for example, a 12-month service billed in advance).
For an AR evaluation, include:
- A deposit covering several invoices with a clear remittance.
- A short payment with no explanation.
- A deposit with no usable reference.
- A split remittance across two customer accounts.
For a bill, “correct” means supplier, entity, line amounts, coding, period, and the source document attached. For a receipt, it means payer, currency, invoice allocations, and the unresolved amount left open. NIST’s AI Risk Management Framework Playbook recommends testing under conditions that match deployment and using simulations before production tests (NIST AI Risk Management Framework Playbook). For a finance pilot, that means records that look like what your reviewers actually see, including the errors they have caught before.
Keep the test away from live writes and customer messages until later.
Step 3: Run the same script with every vendor
Give each vendor the same sample and the same approval rules. Ask them to do these things live, in this order:
| Area | Ask the vendor to demonstrate |
|---|---|
| Source evidence | Open the document behind an extracted field or proposed match |
| Exception routing | Send one difficult case to the person who can resolve it |
| Items left open | Show what it declined to decide, and why |
| Permissions | What the tool can read, suggest, or write, and who approves each action |
| Posting and retry | Reconcile the result to the ERP and show a retry will not create a second record |
| Review effort | Let your operator finish the case and record what work remained |
Pay close attention to the cases the tool should leave open. A tool that confidently applies the unexplained short payment to the oldest invoice has made a decision your AR lead should have made.
Compare with what your ERP already does
Many ERPs already handle part of this. Microsoft, for example, documents automatic receipt matching and automatic workflow submission for imported invoices in Dynamics 365 Finance (Microsoft’s invoice workflow documentation). Ask your ERP partner which steps your current setup performs, then ask each vendor exactly what it adds, so you are not paying twice for matching you already own.
Step 4: Test what the tool is allowed to change
Coding a supplier invoice is one decision. Giving the same tool permission to create the bill in the ledger is another. Evaluate each level of authority you plan to grant.
- Read. The tool names the records it used and the period, entity, and currency.
- Suggest. The proposal shows current and new values and waits for a reviewer.
- Write. The tool’s role is limited to the actions you approved, and the saved record matches the approval.
- Recover. After a failure, a reviewer can see what succeeded, what failed, and what to fix.
Before enabling writes, simulate a timeout and confirm the retry checks for the existing record. Include a bill in a locked period and a supplier with similar names across entities. NIST’s playbook also recommends recording human oversight and keeping logs that help teams investigate errors. Test that directly: ask someone who did not run the pilot to reconstruct one decision from the saved records alone. Audit trails and controls for automated AP and AR lists what that record should contain.
Step 5: Ask how fresh the data really is
Vendors often say “real-time.” Ask what that means for a specific decision. At 10 a.m., treasury may see a customer deposit in the bank feed while collections still has the invoice on its call list. Both views can be current; AR just has not applied the cash yet.
| Source event | What the team can see | What still needs confirmation |
|---|---|---|
| Supplier invoice reaches the inbox | A document to review | Amount, coding, approval, whether a bill already exists |
| Bill receives final approval | It passed workflow | Whether the ERP write succeeded and payment was released |
| Bank feed shows a deposit | The bank reported a receipt | Which invoices it covers and whether AR applied it |
| ERP records a customer payment | A ledger payment record | Whether the bank entry clears and the application is right |
Ask the vendor to show the source and last successful refresh beside each status, and to keep event time separate from the time the system received it. Then ask what happens when a notification is missed, duplicated, or arrives out of order. Intuit, for example, recommends using the QuickBooks Online ChangeDataCapture endpoint to recover potentially missed webhook events from the last successfully processed event (Intuit’s webhook best practices). Have the vendor replay a missed update in a sandbox and show that a repeated notice leaves one record.
A scheduled sync is fine when the decision happens on a fixed cadence and the data is reconciled before use. Pay for faster updates only where the delay would change a decision.
Step 6: Score and price the result
Score each vendor per test case, not overall. A simple 0–2 scale works: 0 for wrong or silently decided, 1 for right but missing evidence, 2 for right with source shown or correctly left open. Then measure time.
Worked example (illustrative)
Suppose your AP sample has 40 invoices and your team’s baseline is 6 minutes of hands-on time each, or 240 minutes. Two vendors run the same sample.
| Vendor A | Vendor B | |
|---|---|---|
| Drafts accepted with light review | 28 at 1.5 min = 42 min | 34 at 2 min = 68 min |
| Exceptions handled by AP | 12 at 9 min = 108 min | 6 at 12 min = 72 min |
| Total AP time | 150 min | 140 min |
| Time saved vs. 240 min | 90 min (2.25 min/invoice) | 100 min (2.5 min/invoice) |
| Errors that would have reached the ledger | 0 | 2 (wrong entity) |
At 1,500 invoices a month, Vendor A saves 1,500 × 2.25 = 3,375 minutes, or 56.25 hours. Vendor B saves 1,500 × 2.5 = 3,750 minutes, or 62.5 hours. At an internal cost of $40 an hour, that is $2,250 versus $2,500 of capacity a month.
Vendor B looks better until you weigh the two wrong-entity bills. In a sample of 40, that is a 5% rate of errors that would post, and each one means a correcting entry, possibly an intercompany adjustment, and a supplier conversation. Ask Vendor B to rerun the sample after configuration before you decide.
Keep two numbers apart: cash you can actually avoid (temp staff, overtime, a planned hire, late fees) and staff time that shifts to other work. Only the first reduces spend. The AP ROI calculator and the AP automation pricing and ROI guide help structure the model; for receivables, use AR automation pricing and ROI.
Price the full workflow
Ask how the vendor prices reads, writes, seats, entities, and exceptions. Add the internal effort to maintain access and fix records. If the tool drafts a result but staff still have to find the PO, count that lookup.
Step 7: Put the tested scope in the contract
Use the demo result to define which records, entities, and workflows the quote covers. Write down who owns setup, ongoing support, and exceptions after go-live, and what the connection can write. Confirm the supported entry method: a file prepared for import, a draft someone reviews, and a posted bill each leave different work for AP.
Then repeat the test on a fresh sample before expanding. The AP automation implementation plan covers the rollout after signature, and AI agents vs. RPA vs. outsourcing helps if you are still choosing between approaches.
Before you sign
- Write down what “correct” means for each record in the sample before you see any vendor’s output.
- Count errors that would have posted alongside minutes saved, and get a rerun from any vendor that had them.
- Put only avoidable cash costs (temp staff, overtime, a planned hire, late fees) in the business case.
- Name the records, entities, write permissions, and exception owners in the agreement.