Hewbers workflow states and bounded review retries

Suppose an implementation stage fails and the run has to wait for another attempt. Sending RetryDue while it’s waiting should let the work continue, but sending the same event after completion shouldn’t start anything again. We can try both timings without starting a real coding job, by passing the events through the lifecycle machine that Hewbers uses for its runs.

Hewbers implements the Goobers stage contract for software work that passes through implementation and review. The small program here uses its actual RunLifecycle declaration and gate decision functions, with synthetic events and decisions in place of a model harness or provider calls.

Start with these three rules from the complete machine:

on StageFailed: Running => RetryPending;
on RetryDue: RetryPending => Running;
on GateCompleted: Running => Completed;

After StageFailed, the value is at RetryPending. Nothing in that transition says the wait has elapsed, so the driver supplies a separate RetryDue event before stepping back into Running. It then sends GateCompleted, which takes this trace to Completed. Here’s that sequence going through the machine, with the state printed after each call:

var run: RunLifecycle = .Running;
println("run=" + name(run));
let events: Vec<RunLifecycle.Event> = [.StageFailed, .RetryDue, .GateCompleted];
for event in events {
    let step = run.step(event);
    assert(match step.disposition { .Taken => true, .Ignored => false });
    println("run=" + name(run));
}

Calling .step updates the machine value in place and returns a disposition, so the loop can check that each event actually took a transition. When it reaches Completed, the driver sends one more RetryDue. There is no matching rule at that point; default { state } leaves the value at Completed, and the disposition is .Ignored.

The diagram adds two other events from Running: AttemptsExhausted goes to Failed, while GateEscalate goes to Escalated. It’s a subset of the generated diagram; the full declaration is bundled with the example, and hew machine diagram src/runner/run_machine.hew can render all its rules:

Waiting for approval gives us a separate timeout case to try. A gate is a checkpoint whose result allows, rejects or defers the next stage, and a missing approval needs an attempt budget just as a failed implementation does. For this example, two attempts are allowed and the final timeout action is escalate.

The gate APIs validate the current and maximum attempt before reaching this shared timeout function:

fn timeout_decision(kind: string, attempt: i64, max_attempts: i64, on_timeout: string) -> Result<GateDecision, string> {
    if attempt < max_attempts {
        return .Ok(GateDecision {
            kind: kind, outcome: "retry", retry: true,
            terminal: false, reason: "gate timed out"
        });
    }
    if on_timeout == "escalate" {
        return .Ok(decided(kind, "escalate"));
    }
    if on_timeout == "fail" {
        return .Ok(decided(kind, "fail"));
    }
    .Err("gate onTimeout must be 'fail' or 'escalate'")
}

On attempt one of two, the first branch returns a GateDecision with retry=true and terminal=false. On attempt two, it falls through to the declared final action. decided produces the final decision with retry=false and terminal=true; changing the action to fail would use the other supported branch.

For the timeout case, the driver calls the human gate API twice and inspects the returned decisions:

let first = gates.human("", 1, 2, true, "escalate")
    .expect("first timeout");
let second = gates.human("", 2, 2, true, "escalate")
    .expect("last timeout");

The integers are the current and maximum attempt, and true says the gate timed out. The empty approval string is intentional because these calls are about the missing response.

terminal at the second timeout is easy to confuse with Completed from the earlier trace. It means this gate evaluation has finished with an escalation, while the machine has a distinct Escalated state for that outcome. These calls return decisions; taking the pictured escalation branch from Running still requires a separate GateEscalate event through the machine.

An approval doesn’t bypass the attempt bound, either. The driver asks the reviewer gate to approve attempt three of two and gets an error before that decision can be accepted. Unknown decisions also return errors.

While a review is waiting, the branch can move as well. A branch’s head is its current commit, and the automated gate compares an actual value with the expected one. The driver uses changed-head and reviewed-head as stand-ins for those values and checks that the mismatch fails.

The runnable example (download ZIP) includes the complete lifecycle declaration, the extracted gate functions and the driver. The older source punctuation was updated with Hew’s supported fmt --migrate command, as recorded in the example’s README. On Linux x86-64, with Hew 0.6.0-rc7, run:

cd examples/hewbers
hew --version
hew build src/main.hew -o lifecycle-demo
./lifecycle-demo

The output is:

run=Running
run=RetryPending
run=Running
run=Completed
retry after completion: ignored, run=Completed
timeout 1/2: outcome=retry retry=true terminal=false
timeout 2/2: outcome=escalate retry=false terminal=true
attempt 3/2: gate attempt is outside its declared retry bound
review approved: pass; changed head: fail; invalid decisions: rejected

The driver checks the dispositions, gate outcomes and attempt bounds, including zero or exceeded bounds and an invalid final timeout action.

A retry becomes more involved when the runner stops after making a provider request. Suppose a merge was accepted, but its acknowledgement was lost before the run could record the result. The Hewbers daemon defaults to actor mode, with process-per-slot supervision also available; in that configuration, the shell supervisor runs a native process for each configured slot.

After a restart, the resumed workflow checks the provider’s merge result against the journalled attempt before deciding what remains to send. If the merge has already succeeded, it can continue with the issue closeout.