The sync between two systems that nobody is watching

What it looks like

A business runs two systems that are supposed to talk to each other: the one where the work happens and the one where the money is recorded. A connection was set up once, probably by whoever sold one of them, and it has run ever since.

Most of the time it works. That is what makes this different from most operational problems. There is no pile of paperwork and nobody is typing. The work is invisible because the failure is invisible.

Then something does not go across: a record with a character the other system will not accept, a customer who exists in one and not the other, a correction made after the transfer. The connection sets it aside and carries on.

The question worth asking is: when something fails to go across, who finds out?

What it costs

The hunt is unplanned. Finding one bad record inside a month of transactions takes hours and someone senior, and it happens while the books are being closed and everything else is already due.

The corrections get retyped. Where a connection runs one direction only, a fix made in the second system has to be made again by hand in the first, or the two drift apart. That retyping is a recurring manual task hiding inside something everyone calls automatic.

Where it goes wrong

• Nobody owns the exception list. The connection is often already setting failures aside somewhere: a queue, a log, a notification to an address nobody reads. The report existing is not the same thing as the report being read.
• The setup was a one-time event. Whoever configured it has moved on, and nothing has been examined since.
• One direction is assumed to be two. Whether corrections flow back only surfaces the first time something needs correcting.
• Both sides changed underneath it. Software updates on its own schedule, and a connection that was right three years ago may be doing something different now.

What a better version looks like

A standing check that compares the two records that are supposed to agree, on a schedule, and reports every difference while changing nothing.

Changing nothing is the design. Something that reaches into the books and fixes what it believes is wrong is a liability. Something that says these two disagree, here is the evidence, you decide is useful immediately and cannot do damage. A flag means the records disagree; it does not say which one is right.

A difference then has to reach a person with a name and a regular time to look at it. An exception queue nobody owns is where this started.

Questions worth asking about your own operation

• If a record failed to transfer last Tuesday, what would have happened, and who would have seen it?
• Does the connection log its failures, and does a named person read the log?
• Do corrections made in one system flow back to the other?
• When was the connection last examined?