Case study

From batch jobs to near-live student data

Batch jobs moved student data between systems on a schedule, so the systems around the student record could be behind it. The fix was an integration layer that moves each change when it happens.

Consulting engagement about 2 min read

The situation

The student information system was the system of record, and the systems around it were fed by batch processes. A change made during the day reached the systems downstream after the next batch had run.

In my experience a delay like that has a cost that does not show up in a diagram. People stop trusting the second screen and start checking both. And when a scheduled job fails, the first sign is often a question from someone who noticed.

What I built

An integration layer that keeps student data in step across systems as changes happen.

  • Events in place of schedules. A change in the student system raises an event, and the integration layer carries it to the systems that need it.
  • Serverless on Azure. Functions that run when there is work to do, with no servers to patch.
  • The supported interfaces only. Reads and writes go through the student system’s own APIs, which is the route the vendor supports through upgrades.
  • Bulk loads handled separately. Large batches of data, which events are the wrong tool for, move through SSIS, a separate bulk-loading tool.
Event-driven sync from the student information system A change in the student information system raises an event. The event is placed on a queue. A serverless function picks it up and delivers it through each target system's interface. A log records every message, and failures are retried and reported. Student system a record changes EVENT Queue held until delivered SERVERLESS Function runs on demand Learning platform Finance Reporting Message log every message recorded; failures retried, then reported to a person
Event-driven sync from the student information system A change in the student information system raises an event. The event is placed on a queue. A serverless function picks it up and delivers it through each target system's interface. A log records every message, and failures are retried and reported. Student system a record changes EVENT Queue held until delivered SERVERLESS Function runs on demand Learning Finance Reporting Message log every message recorded; failures retried, reported
A change is captured once, queued, and delivered to each system that needs it.

Safeguards

Moving data faster only helps if it is still right. My standard for a layer like this is the same every time: every message is logged, delivery can be retried without creating a duplicate, and anything that cannot be delivered is reported to a person instead of being lost.

The result

Batch processes were replaced with near-live data movement, so the systems around the student record keep up with it during the working day.

What I would tell another school

Do not start with the technology. Start by listing every place a staff member retypes something or waits for tomorrow. Those are your integrations, in priority order.

Printed from https://www.datazak.com/work/real-time-student-data-sync/. An address in parentheses after a link that begins with a slash is a page on https://www.datazak.com.

More work

Full-time architect role, then a consulting engagement

A report that documents every workflow you are running

A self-service report that lists, classifies and explains every production workflow in the student system, so nobody has to remember what is running.

Full-time architect role

A financial aid migration you can run twice

A check-only report is read and approved first; then one command corrects financial aid records for a whole student population, can be run again safely, and leaves before-and-after evidence for every student.

Have a system that needs untangling?

Tell me what you run and where it hurts. The first call is 30 minutes, free, and you will leave with a clear next step whether or not we work together.

Search this site

Up and Down move, Enter opens the result, Esc closes.