Kubernetes: CronJobs should support timezones

@iterion There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

_Note: method (1) will trigger a notification to the team. You can find the team list here and label list here_

/sig apps

Prefer the second option as it's looks more nature to me

@soltysh @kubernetes/sig-apps-feature-requests

I also like the second form, but it seems that the easiest path to implementation would be to strip that time off and use it separately. Also, an argument could be made, that, since we're using the "standard" cron template we shouldn't mess with that.

It seems like this should be straightforward to implement:
Currently, we just pass time.Now() into the syncOne loop. The cron lib that k8s is using seems to work by just using times in the zone you care about. So, we could pull the timezone from the CronJob and pass that time with zone into the syncOne func.

//init the loc, needs error handling
//perhaps default to local if Timezone was invalid
loc, _ := time.LoadLocation(cronjob.Spec.Timezone)

//set timezone,  
now := time.Now().In(loc)

Also, I'm happy to try to implement this.

I'm not the right person to review your PR, but I do have a few questions about semantics.

• What are the semantics when no timezone is specified?
• Does the implementation depend on the timezone being correctly configured on the controller host?
• When do we validate timezone legality? At object create/update time, or later?
• What happens if a timezone name was legal when the API object was created, but in the future no longer exists? This could happen because a name is removed (maybe a city or nation ceases to exist), or because the underlying tzdata package has changed to an older version.

If no timezone is defined it would default to the current behavior which would be to use the timezone of the host. I would guess in most cases this would be configured as UTC, but a user could really choose any timezone. As such, there is no greater dependency on timezone being correctly configured on the host as a result of this PR.

I currently just bail out and use the host's timezone if the user specified something invalid. Here, I'm leaning on golang's time.LoadLocation(str) method. Any string accepted by that function would be considered valid. I'm not currently validating this at create/update time, though it seems that would be a pretty easy thing to do.

As far as the last question, that one's a bit more undefined. Currently, and probably in the future, it would most likely fall back to scheduling the job based on the default time zone. I'm open to suggestions on how to handle this case.

What happens if a timezone name was legal when the API object was created, but in the future no longer exists? This could happen because a name is removed (maybe a city or nation ceases to exist), or because the underlying tzdata package has changed to an older version.

My suggestion for that would be to fail job creation with an appropriate event. But frankly speaking I doubt this happens that often. I'll make sure to have this added in your implementation @iterion.

and cron format not support year filed, how to run a job only once at a specified time?

:+1:

@iterion Thanks for the issue and pull request. We discussed it quite a bit over the past week, both in SIG Apps and in SIG Architecture. While the use cases this could be useful for make sense we also had to discuss the downsides. For example, every compliant Kubernetes distribution would need to have a timezone database and keep it up to date. Timezones, daylight savings time, and the details of making this work while the real world details can and do shift is hard. This would be additional work for the more than 20 distributions that exist today.

At the same time there has been a shift in Kubernetes development. I'm not sure how widely communicated it has been (the communication was another point discussed in SIG Architecture). The shift is a desire to have core Kubernetes focus on stability and being slow moving. People are now encouraged to innovate and solve these kinds of problems in the ecosystem rather than core.

Instead of putting this in Kubernetes the ask is to:

1. Develop this in the ecosystem (e.g., a controller) that others can use. Distribute it, solve the problems there, and see what update looks like
2. If the solution is widely adopted and can be used by everyone (including small scale, multi-cluster, etc) then it could be considered for core Kubernetes

If it helps, this isn't the only features that's receiving this same direction (even in the same SIG Architecture meeting). I'm starting to think of this in a similar vein to something like wordpress (I use wordpress because it's a very common example). There difference between it being plugin or part of wordpress itself. The current direction is to encourage innovation in plugins, so to speak.

If you build something like this we'd like to have it demo'd at SIG Apps to showcase it for folks. It could also be demo'd at the community meeting.

If you have questions on this please feel free to reach out to me.

Given the previous comment I feel like we can close this issue. Feel free to reopen if you don't agree.

FWIW, "how to set the timezone" is asked by every single of my users that makes a CronJob (developers and admins alike).

@iterion What did you end up doing about this? We're considering various options for our use case, including

• running from a fork of kubernetes with your fix
• Copying the cron controller code into an external controller, and using a different apiVersion to distinguish the two.
• Building something specific to our project in Ruby which mirrors the kube cronjob controller implementation but based on an app-specific config file rather than CronJob resources.

If you already have something that's working for you that we could use as a starting point, or if we could collaborate on something that could serve to encourage @mattfarina to reverse his decision not to permit including this (which dramatically harms the adoption of this kube feature), that would be awesome.

I've actually switched to a new job, so the original problem doesn't exist for me, personally, anymore. As far as I know, my former colleagues are just dealing with the pain that results from it.

That said, if I had the time to devote to this I would likely just write a small controller that would watch and modify CronJobs as needed. I'd likely just add an annotation to CronJobs with the desired time and time zone and have the controller cast that schedule to UTC and modify the CronJobs. You could just run this sync loop at an interval to catch the problem edges, i.e. when daylight savings starts or ends. In this way, the controller would just be a very slim wrapper on top of the already existing CronJob API.

encourage @mattfarina to reverse his decision not to permit including this

To make it clear, this decision was done collectively during one of sig-architecture meetings. Matt was only the person expressing that decision in this issue.

The problem of timezones with all bells and whistles has long been solved in the Linux world. Why is this that much of a prerequisite for a Kubernetes deployment?

It is difficult to discuss about the significance of use cases I admit. However (and in my humble opinion), being able to schedule services in sync with the services' users seems important enough to me to be eligible for Kubernetes core.

Would anyone be willing to write an operator with a TimezoneAwareCronjobController? Most of the code written by @iterion can probably be recycled.
Moving TZ handling into a packaged application would solve the distribution argument that killed first class cronjob support in Kubernetes.

FWIW, https://gopkg.in/robfig/cron.v2 (which the existing controller uses am older subset of), does support passing in TZ, and a slight richer set of time definitions.
We have a controller internally that creates cron jobs for some of our internal workloads, and it is an outstanding feature request to be able to specifcy timezone for those. I need to work out how to safely update an existing cronjob definition without accidentally skipping, or rerunning jobs. At this point it feels like it might actually be easier to steal the job scheduling from the exiting contoller and create job definitions directly (rather than cronjobs).

Since I've started using K8s I come back to this issue twice a year, when I'm forced to change the schedule on all the CronJobs, hoping that it's finally been reopened. If chronos and crontab have the feature, how is K8s still not supporting this?

long-term-issue (note to self)

Hopefully this problem will be reopened and it's a little strange to look at UTC time every day.

I have a cronjob that needs to run on the first day of each month at local time, but UTC time is 16:00 on the last day of last month. How do I define the last day of last month? Cronjob does not support L.

It appears that an alternate solution does not exist. See my questions here and here

For anyone still waiting for this; I created a controller derived from the native Kubernetes cronjob controller and the original PR that was closed.

https://github.com/hiddeco/cronjobber

Edit (2019-05-12): thanks to @mterron, Cronjobber now supports a host independent timezone database using a sidecar.

@hiddeco would it be better to have a controller control standard CJ controller instead of modifying existing one? With the latter approach one needs to keep up with the actual CJ controller changes, which may not be always feasible. Thank you for your work.

@andrewsav that would be preferable but is not doable. The only way I see to ‘control’ the existing CJ controller is by modifying the schedule of a CronJob to the offset of a time zone. There is no way you can do that for all possible schedule combinations.

I do not expect that many changes to the CJ controller anyway, plus we are not obligated to stay in sync with it to keep it running as it is self-contained (except for dependencies on the client and Job specifications).

@hiddeco

There is no way you can do that for all possible schedule combinations.

Can you explain this please?

@AndrewSav simple example: 0 0 1 * * running in NZDT (UTC+13) would mean you would need to recalculate it every month because you will need to run in on the last day of the previous month at 11AM in UTC.

If the complexity of your schedule goes up the amount of changes you need to make to keep it in sync goes up too. Plus you would need to change it every time a time zone area changes from DST to ST.

This makes it extremely complex and opens up a lot of space to run it at the wrong time or maybe even worse; not at all.

A solution is to run a cron every "possible" hour depending on your timezone and skip it if the current time is not the expected one. I wish I could do it at the kubernetes level though.

For DST this means it will run twice a day rather than once and skip the first one or the second one depending on the DST.

For example, every compliant Kubernetes distribution would need to have a timezone database and keep it up to date. Timezones, daylight savings time, and the details of making this work while the real world details can and do shift is hard. This would be additional work for the more than 20 distributions that exist today.

Timezone databases, DSTs, are very well maintained by OS distributions, which kubernetes distributions should rely on. The "it's hard work for the distributions" excuse is not valid IMO.

I'd not argue if the concept was called PeriodicJob or RepeatingJob. But instead, it is called CronJob. It is tied with calendar and the clock. There is no denying that timezone awareness is a critical requirement for this.

Another solution; maybe the problem can be solved by removing CronJob completely from kubernetes. Not that it would help anybody.

Another solution; maybe the problem can be solved by removing CronJob completely from kubernetes. Not that it would help anybody.

Agreed, if they can't implement a proper solution in k8s they shouldn't add in the solution at all. Cronjob's are designed to work with timezones, also all OS's today handle timezones transparently with the associated OS functions.

The reasoning does not appear to be coherent, if we are going to go by this logic then there should be a lot of features that exist in K8s which shouldn't exist.

This is indeed a bug. The name Cronjob itself is misleading since it does not follow the Cron specification. Most kubernetes users will assume Time Zone support comes out of the box. It is dissapointing that the core developers chose not to include this. This issue should affect > 90% of the users who don't operate on UTC time.

I'd urge to revisit this. One problem I am having is that I want certain jobs to run at 1am on the first of the month in Japan--certainly a reasonable/standard use case. Given that cron is defined in UTC there is no way for me to specify this; 0 0 1 * * will run at 9am Japan time. (It would also be nice to add 0 0 L * * for last day of month which is used in some cron implementations)

@johnnyshields you should probably ask @soltysh to reopen.

While I'd love this to be re-considered, I think the chances are slim. The team got together, discussed it and decided that this would cause too much additional work. This is not something that has changed since then, so the decision has not likely changed either.

Just take a look (and contribute!) to CronJobber https://github.com/hiddeco/cronjobber. It does what everyone wants.

A PR rebased on the latest version of the CronJob controller would be great but it works perfectly as it is.

While I'd love this to be re-considered, I think the chances are slim. The team got together, discussed it and decided that this would cause too much additional work. This is not something that has changed since then, so the decision has not likely changed either.

They should then at least be consistent and remove the current implementation of cronjob then because its very misleading (to say the least).

@mterron the problem with that one, is it was patched almost a year a go and has not been updated since, in the meanwhile we had a dozen of point releases of kubernetes with improvements and security fixes. So one cannot be confident running the stale contorller that they are getting advantage of all the latest changes and feature in kubernetes in this area.

@AndrewSav and that's why I was asking for help with it. I've done everything I can. I'm not sure there's any security risk with it though, Kubernetes changes all the time but it doesn't mean that the change is meaningful in a security sense.

Having said that, a rebase would be cool, feel free to contribute and help with it.

@mterron you are the author of that one aren't you? So given that even you cannot find time to do that, it is not likely that people would contribute into something that is no longer active. Would be extremely nice though, I agree. When that project was first pushed, I briefly considered if I gamble on it and put it into my clusters, and then I thought that chances of it being maintained are not great. Now a year later I see I was not wrong. So I thank you for your work, but I'm afraid in it's current form it's not sustainable.

If it takes off though and someone keep consistently rebasing and maintaining it that would be just wonderful.

No, I'm not the author. I'm just a user that contributes to projects I use when I can.
I understand where you are coming from but I think your expectations are a little high for something people do in their spare time for the benefit of the community.

@mterron I think I'm on the level with my expectations. If anything, they are low. That's the main reason I refrained from using the linked work in my clustres.

@AndrewSav you should do a diff between cronjobber and the upstream cronjob controller. Last time I did (1.17) there was no difference. The upstream cronjob controller is effectively unmaintained by your definition.

The upstream controller also has horrific performance characteristics, which of course cronjobber inherits.

@benlangfeld

you should do a diff between cronjobber and the upstream cronjob controller

It's not as much about me, it's about it being sustainable for general use. It's hardly reasonable to request every potential user to do a diff. Even if it's a fact that there were no chages, which I'd like to examine further below.

The upstream cronjob controller is effectively unmaintained by your definition.

This is not a fair interpretation of what I said.

On the subject of diffs: what are you diffing with what? There are over 50 files in that repo, and not all names match the one from the kubernetes sources. Can you explain how can we convince ourself that there were no changes? Preferably a bit more detailed that in your previous comment, as it's not obvious. It particular, which files on which branches / commits on each side have you compared, and detected no chages?

Given the intensive participation in this issue, I think it is fair to request for a re-consideration for a Kubernetes core feature.

I find the following statement above important: “If the solution is widely adopted and can be used by everyone (including small scale, multi-cluster, etc) then it could be considered for core Kubernetes”. There seem to be solutions, maybe not yet fully ready. So, a statement of the core team that the use case is acknowledged to be eligible for K8s core would be a great signal to the developers of these solutions to make them ready for a pull request.

@AndrewSav as the author of the 'fork' I have to respectfully disagree with anything you have stated thus far, and as a maintainer of multiple OSS projects I am surprised by your attitude.

First, cronjobber was simply derived from the Kubernetes CronJob controller because it was the most straight forward fix, requiring the least amount of time for a mature (and proven) solution. Given the reasons stated below, I did not expect the upstream to change much (nor has it).

Second, spawning job X at time Y while taking into account timezone Z isn't rocket science, the specification for the concept time has not changed in years, and shouldn't require any maintenance other than ensuring it can still spawn Kubernetes native jobs at the right time (and the time zone databases are up-to-date, for which there is a helper available thanks to @mterron).

As long as it is able to spawn these jobs, which it is perfectly capable of last time I checked, my time is better spent elsewhere. If this does not meet your standards, I would suggest hiring someone to write and maintain a likewise solution for you.

@hiddeco

I have to respectfully disagree with anything you have stated thus far

Anything? That's a strong statement.

Given the reasons stated below, I did not expect the upstream to change much (nor has it)

It might not change _much_ but it also won't remain static. The kubernetes team stated that they want a "widely adopted" solution before it "could be considered for core Kubernetes". In my experience solutions that are not being actively maintained (no changes for almost a year) rarely get "widely adopted". In addition, nobody so far demonstrated, that Kubernetes CronJob controller and its dependencies has not changed at all. I personally cannot figure out a good way of comparing, all files that was affected and all the underlying dependencies.

Second, spawning job X at time Y while taking into account timezone Z isn't rocket science, the specification for the concept time has not changed in years, and shouldn't require any maintenance other than ensuring it can still spawn Kubernetes native jobs at the right time (and the time zone databases are up-to-date, for which there is a helper available thanks to @mterron).

This sounds overly optimistic to me. This is rarely about "specification" changes. It's small things that kills you. Bugs are found constantly, even in "mature and time proven" projects and get fixed. Spawning a job maybe not a rocket science, but finding out how related changes in kubernetes codebase affect your changes can get tricky fast. For example, during last year efforts was made of moving client api (kubectl) to staging. Does it affect cron jobber? I don't know. The cron library version used was updated upstream. Does it affect cron jobber? Again, I don't know.

As long as it is able to spawn these jobs, which it is perfectly capable of last time I checked, my time is better spent elsewhere.

And that's fine. You've already done more that you had to. You merged the PR, and you published your work for anyone to use. Thank you.

If this does not meet your standards, I would suggest hiring someone to write and maintain a likewise solution for you.

Again, this is not about me, as I explained in a comment above. It's about being useful for a wider class of people. It seems like I'm repeating myself now. May be everything to say on the topic was already said? What do you think?

This sounds overly optimistic to me. This is rarely about "specification" changes. It's small things that kills you. Bugs are found constantly, even in "mature and time proven" projects and get fixed. Spawning a job maybe not a rocket science, but finding out how related changes in kubernetes codebase affect your changes can get tricky fast. For example, during last year efforts was made of moving client api (kubectl) to staging. Does it affect cron jobber? I don't know. The cron library version used was updated upstream. Does it affect cron jobber? Again, I don't know.

Honestly this comes to me as whatabouttism thats resulting in obstructionism. This logic can be used to argue against anything so I have no idea what you are trying to achieve here. There is nothing really that complex about running cronjobs that are timezone aware. Any mainstream *nix system (which is what k8's runs on) has had rock solid timezone support for decades (because surprise surprise, *nix systems have been predominantly running in server environments which has similar set of concerns concerns for 40+ years).

Again, this is not about me, as I explained in a comment above. It's about being useful for a wider class of people. It seems like I'm repeating myself now. May be everything to say on the topic was already said? What do you think?

And as has been stated ad-nauseam, the current implementation is close to useless because it completely ignores timezone where as the implementation that is timezone aware is much more useful to a wider set of people, whats your point here?

This logic can be used to argue against anything

I don't think it can. You are welcome to demonstrate it, if you like, but please stick to what's said, not what could have been implied.

so I have no idea what you are trying to achieve here.

Oh, I'm just answering the comment from hiddeco directed at me, you can find it above.

And as has been stated ad-nauseam, the current implementation is close to useless because it completely ignores timezone where as the implementation that is timezone aware is much more useful to a wider set of people

If we rephrase that, to say that you and I have a preference that kubernetes cronjob supported time zones, that would be right. I find "closs to useless" a bit too strong, but I strongly agree that a supported timezone aware implementation would be mich more useful.

whats your point here?

The point is that there is no evidence, that the "fork" we are discussing is "widely adopted", and therefore it's unlikely that the kubernetes team will take it on board, which is a real pity, because that's what I would love to see incorporated into kubernetes.

Bugs are found constantly, even in "mature and time proven" projects and get fixed.

First, "mature" and "time proven" words has no place in a sentence about kubernetes, or its ecosystem.

Second, since when bugs are a problem in kubernetes? @fejta-bot will just "fix" (close) them automatically. No need to think about bugs. All is well.

I wonder when/if any maintainer will care/comment?

I think that this should be documented under https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations

I agree that "cronjobs" on kubernetes should have time-zone support because Kubernetes clusters run on Linux which provides this. Are we rejecting this notion because some party has designs on running Kubernetes (or variant) on Windows in the distant future? Pretty weak reason to reject this issue if the case. Please can we at least add the "triage/unresolved" tag to distinguish this from issues that have been solved?

If the decision is that core k8s should not extend to this functionality, then I agree with the comment above that the cronjob support should be removed so that people are encouraged to adopt an ecosystem solution as they choose. This really sticks out as something in core k8s that is not explicit about the intended result, and that trips people up.

Since Go 1.15 now has got a time/tzdata package in its core library, possibly …?

How am I supposed to over-provision my cluster at 7 in the morning to 7 in the evening?
Where my users are, there is summer time and winter time.
So I cannot rely on that at all.

To NOT manage timezone with cronjobs was a very strange decision you took.

We found a good workaround for this issue: stop using K8s CronJobs and use Apache Airflow instead.

Life has gotten much better ever since we did. Join us!

This issue is solved externally there: https://github.com/hiddeco/cronjobber

@mattfarina

1. If the solution is widely adopted and can be used by everyone (including small scale, multi-cluster, etc) then it could be considered for core Kubernetes

How, and when, is the adoption of the community solution measured?

https://github.com/kubernetes/enhancements/pull/1996

@mattfarina @soltysh now with KEP-19: Graduate CronJob to stable, which mention timezone as one of the improvements for consideration, is approved and merged and the work on cronjob controller v2 is in process, can this issue reopen and reconsidered?

It mentions this as a consideration/possible improvement. It doesn't guarantee if nor when this will be approached. I can say with 100% guarantee it's not going to happen before CronJob GA, which will take this and another 2 releases.

@soltysh It can still be properly discussed. It has been widely demonstrated that this is a necessary feature. The insistence on kicking the ball down the road on this shows an enormous disconnect between the maintainers and users.

@benlangfeld I'm very open to this, but time is the main limiting factor here :disappointed:

@benlangfeld I'm very open to this, but time is the main limiting factor here 😞

There is already a proposed fix. What input do you need in order to get this accepted?

The controller is being re-written, that's the primary focus right now. The change will be possible only with the new controller. It's pointless to change the old one which is meant to be replaced.