Flynn: controller,scheduler: Fix job event duplication constraint

Created on 9 Sep 2016 · 4Comments · Source: flynn/flynn

We currently only create a single event per job ID and state (see here), but it is feasible for a job to enter a particular state twice.

As an example, if a host is deemed unhealthy by the scheduler, all jobs on that host are marked as down, but then if the host comes back the jobs may still be running so the jobs will re-enter the running state. Subsequent deploys involving those jobs will then fail because down events won't be emitted (the down event already exists from when the host became unhealthy).

Couple of options:

drop the unique constraint (need to think about whether anything assumes the events won't be duplicated)
when the scheduler marks jobs as down, just stop them if they appear running again since replacement jobs have already been started

@titanous @josephglanville @jvatic thoughts?

componencontroller componenscheduler kinbug

Source

lmars

Most helpful comment

Alternatively a different state could be added for jobs that are gone because the host went away.

titanous on 9 Sep 2016

👍2

All 4 comments

when the scheduler marks jobs as down, just stop them if they appear running again since replacement jobs have already been started.

I think that's the correct response.

titanous on 9 Sep 2016

👍1

Alternatively a different state could be added for jobs that are gone because the host went away.

titanous on 9 Sep 2016

👍2

@titanous I don't think adding a new state for jobs on disconnected hosts will really help.

When the scheduler reconnects to a host whose jobs were previously marked as stopped (or put in some other state like unknown), it still needs to make a decision whether to just stop any jobs which are still running or try and mark them as running again (leading to duplicate up events in the controller).

I think we should just go with stopping the jobs straight away after reconnection.

lmars on 9 Oct 2016

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.

titanous on 18 Feb 2021

Was this page helpful?

0 / 5 - 0 ratings