Consulting engagement
From batch jobs to near-live student data
A new integration layer on Azure replaced batch jobs, so a change in the student system reaches the other systems as it happens.
Case study
The race that emailed students an empty link
Students were being emailed an invitation to an assessment with the link missing. Not always, just often enough to matter. The cause was two automated jobs racing each other.
The situation
When a student reached a certain step, two things had to happen in order: a link had to be fetched from an outside provider and saved to the student’s record, and then an email containing that link had to go out.
The first job ran on a timer, checking the database every few seconds for new records. The email was sent by the student system’s own scheduler. Usually the timer won. Sometimes the scheduler did, and the student received a message with an empty space where the link should have been.
Problems like this are miserable to chase. They rarely show up in testing, they cannot be reproduced on demand, and each one is a confused student and a staff member apologizing.
Further down this page, a small simulation runs that race on twelve made-up students, so you can watch it happen.
What I built
I replaced the timer with a design that does not wait for one.
- An event in place of a timer. Saving the record raises an event right away. Nothing waits for the next check of the database.
- A cloud function does the fetching. It picks the event up, gets the link from the provider, and writes it back to the student record.
- The link is there first. Because the function starts the moment the record is saved, the link is in place ahead of the scheduled email.
- Built to be rebuilt. The cloud resources are defined in code, secrets are kept in a vault, and the function reports on itself, so the next integration starts from a working pattern.
Safeguards
The old timed job was not switched off on day one. It stayed in place as a fallback for a trial period, so that if the new path had a problem the student would still be served. The new path was checked in production with a real message before it was trusted.
The result
Within hours of go-live the same timing came up in production, and the new path had the link in place first. Students got their link. The same pattern was then reused for the next integration, which no longer needed to check on a timer either.
What I would tell another school
If something in your student system “sometimes” goes wrong, suspect timing before you suspect people. Two automated jobs that each work perfectly can still fail together. The fix is to stop one of them waiting on a timer, not to hope the timer wins.
Printed from https://www.datazak.com/work/event-driven-invitations/. An address in parentheses after a link that begins with a slash is a page on https://www.datazak.com.
Watch the race
Two jobs, twelve students, one timer that sometimes loses
The race described above is easier to believe when you can see it. Twelve made-up students go through the same two jobs twice: once with the link job on a timer, once with it started by the record being saved. Pick how often the timed job runs late, then run the race.
Made-up records. Student 1001 to 1012 are not real students and nothing is sent. A simplified model that runs in your browser: each tick is a moment, not a real second.
An example run is drawn below. Pick how often the timed job runs late, then run the race.
Lane A: Timer
Job 1 waits for its next timer check, then fetches the link and saves it. Job 2, the email scheduler, sends at tick 6 whether or not the link is there.
- Student 1001 Link first
- Student 1002 Link first
- Student 1003 Empty link
- Student 1004 Link first
- Student 1005 Link first
- Student 1006 Empty link
- Student 1007 Link first
- Student 1008 Link first
- Student 1009 Link first
- Student 1010 Empty link
- Student 1011 Link first
- Student 1012 Empty link
Example run: 4 of 12 invitations went out with an empty link.
Lane B: Event
Saving the record starts job 1 at once, so the link is saved by tick 2 at the latest. Job 2, the same email scheduler, still sends at tick 6 and finds the link already there. Same students, same fetch times.
- Student 1001 Link first
- Student 1002 Link first
- Student 1003 Link first
- Student 1004 Link first
- Student 1005 Link first
- Student 1006 Link first
- Student 1007 Link first
- Student 1008 Link first
- Student 1009 Link first
- Student 1010 Link first
- Student 1011 Link first
- Student 1012 Link first
Example run: 0 of 12 invitations went out with an empty link. The link was saved ahead of the scheduled email each time.
- Waiting for the timer
- Fetching and saving the link
- The email goes out
The lesson: the fix is to stop one of them waiting on a timer, not to hope the timer wins.
Show the technical detail
Each run draws its delays from a small seeded pseudo-random generator (Mulberry32), so a run can be repeated from its seed: Replay shows the same result, and the three presets reuse the same draws, which is why a student who is late at Rarely stays late at Often. Run again picks a new seed. The presets set the share of students whose timed job runs late: about 15 percent, 35 percent and 65 percent. The bars move with CSS transitions whose lengths come from the same numbers, and with reduced motion switched on in your system the end state is shown at once. The script is plain JavaScript with no library: the model is the first few functions, the drawing is the rest.
More work
Full-time architect role, then a consulting engagement
A report that documents every workflow you are running
A self-service report that lists, classifies and explains every production workflow in the student system, so nobody has to remember what is running.
Consulting engagement
A document migration checked record by record
Student documents moved on deadline with zero defects, because every record was verified before sign-off, not a sample.
Have a system that needs untangling?
Tell me what you run and where it hurts. The first call is 30 minutes, free, and you will leave with a clear next step whether or not we work together.