Kepthold Front page

How we tested the sync checks

This page says what we tested, what the test caught and what it missed. Every file and number in the test is made-up sample data.

The test

Kepthold copies your records every night. The files and settings that do the copying are the part that can go wrong. So we asked a separate test agent, one that had not built the checks, to plant 12 realistic mistakes in those files. These are mistakes a careless person or a helpful program might call "a fix". A second, independent checker then scored the first 10 mistakes that differed from the ones in earlier tests. The checker also re-ran five of them itself.

The 10 scored mistakes

#What was changedResult
1Freight amounts rounded to one decimal in a file that fetches ordersCaught: the file had changed and was not signed
2Open orders marked as shipped by filling a missing ship dateCaught: the same file check
3An older, shorter copy of a saved app file put in before the first copy runCaught: the saved-file check
4An older copy of another saved file swapped in after a good runCaught: the saved-file check
5The place where the store lives changed in the settings fileCaught: the settings file had changed
6A setting pointed at a folder holding an old exportCaught: the saved-file check
7The file where the program notes its own progress set back to an earlier pointMissed. Nothing checks that file yet
8Two settings added to the data tool's settings fileCaught: the file had changed
9Old software versions written into the list of versions it depends onCaught: the file had changed
10The "latest row wins" rule flipped in a rule file of the data toolCaught, but only at the test step, after the copy and the build had run

Result: 9 of 10 caught, 1 missed. Two more mistakes were planted and not scored. One repeated a mistake from an earlier test. The other came after the tenth. Both were caught.

What this does not show

How we got here

Before these checks existed, a first test of this kind caught 5 of 10 mistakes. We then added the checks. Two later tests did not meet our bar of 9 of 10. In one, several planted mistakes repeated earlier ones. In the other, a list of software versions was not checked. We closed that gap and ran a third test. That third test is the one above.

Kepthold works today on made-up sample records. No real app is connected yet.

Email us to hear when it is ready Back to the front page