How we tested the sync checks
This page says what we tested, what the test caught and what it missed. Every file and number in the test is made-up sample data.
The test
Kepthold copies your records every night. The files and settings that do the copying are the part that can go wrong. So we asked a separate test agent, one that had not built the checks, to plant 12 realistic mistakes in those files. These are mistakes a careless person or a helpful program might call "a fix". A second, independent checker then scored the first 10 mistakes that differed from the ones in earlier tests. The checker also re-ran five of them itself.
The 10 scored mistakes
| # | What was changed | Result |
|---|---|---|
| 1 | Freight amounts rounded to one decimal in a file that fetches orders | Caught: the file had changed and was not signed |
| 2 | Open orders marked as shipped by filling a missing ship date | Caught: the same file check |
| 3 | An older, shorter copy of a saved app file put in before the first copy run | Caught: the saved-file check |
| 4 | An older copy of another saved file swapped in after a good run | Caught: the saved-file check |
| 5 | The place where the store lives changed in the settings file | Caught: the settings file had changed |
| 6 | A setting pointed at a folder holding an old export | Caught: the saved-file check |
| 7 | The file where the program notes its own progress set back to an earlier point | Missed. Nothing checks that file yet |
| 8 | Two settings added to the data tool's settings file | Caught: the file had changed |
| 9 | Old software versions written into the list of versions it depends on | Caught: the file had changed |
| 10 | The "latest row wins" rule flipped in a rule file of the data tool | Caught, but only at the test step, after the copy and the build had run |
Result: 9 of 10 caught, 1 missed. Two more mistakes were planted and not scored. One repeated a mistake from an earlier test. The other came after the tenth. Both were caught.
What this does not show
- Six of the nine catches are the same kind: a file changed and nobody signed it. A change that someone signs without reading gets through.
- A few places are not fully guarded yet: a file where the program notes its own progress, a settings file for one of its tools, extra files that add steps, and some computer settings. Some changes there were caught late or not at all.
- A few changes stop only at the test step, after the copy and the build have run.
- It was one tester's set of ideas on made-up data. We did not test real business data or real apps.
- Deciding whether a mistake repeats an earlier one is a judgement call.
How we got here
Before these checks existed, a first test of this kind caught 5 of 10 mistakes. We then added the checks. Two later tests did not meet our bar of 9 of 10. In one, several planted mistakes repeated earlier ones. In the other, a list of software versions was not checked. We closed that gap and ran a third test. That third test is the one above.
Kepthold works today on made-up sample records. No real app is connected yet.