69 lines
2.5 KiB
ReStructuredText
69 lines
2.5 KiB
ReStructuredText
|
|
Scheduled Tasks
|
|
===============
|
|
|
|
By "scheduled task" we mean, a command which runs automatically at a
|
|
pre-determined time. See also :doc:`commands` and :doc:`scripts`.
|
|
|
|
For now, Rattail assumes that any scheduled tasks will be manually
|
|
configured to run via `cron`_. See your operating system's
|
|
documentation for details.
|
|
|
|
.. _cron: https://en.wikipedia.org/wiki/Cron
|
|
|
|
(In the future, Rattail may add a `dedicated scheduler daemon`_, but
|
|
it is not yet a priority.)
|
|
|
|
.. _dedicated scheduler daemon: https://redmine.rattailproject.org/issues/13
|
|
|
|
|
|
Luigi
|
|
-----
|
|
|
|
`Luigi`_ is a "task runner" of sorts. Rattail uses it to help
|
|
orchestrate a "batch" of commands such as is often needed for
|
|
overnight automation (aka. "EOD").
|
|
|
|
.. _Luigi: https://luigi.readthedocs.io/en/stable/
|
|
|
|
The general idea is to maintain a Python module containing all of the
|
|
individual tasks which need to run. Rattail can then invoke the
|
|
module in such a way that Luigi runs all the tasks, keeping track of
|
|
which tasks have (not yet) ran, and in which order they should run
|
|
etc.
|
|
|
|
|
|
Overnight
|
|
~~~~~~~~~
|
|
|
|
Often there is just one "overnight" module, which contains all tasks.
|
|
But in some cases you may want a couple of separate modules, to run at
|
|
different times. A common example is like:
|
|
|
|
* run "all" (typical EOD) tasks, starting at 1am
|
|
* run "backups" for all servers, starting at 4am
|
|
|
|
In this example the "all" task may include a dozen or more individual
|
|
"steps" (which, confusingly, are also called "tasks"). Maybe data is
|
|
exported from one system to another, reports are ran, etc. Then the
|
|
"backups" task runs later, and may itself also contain multiple steps,
|
|
or perhaps just one.
|
|
|
|
There are two main benefits from wiring up overnight tasks via modules
|
|
to be ran by Luigi.
|
|
|
|
The first is that each individual step will (typically) start
|
|
immediately after the previous one completes. This means all steps
|
|
are "packed in" as tightly as possible, to help keep overall run time
|
|
down. Whereas scheduling each step separately often involves
|
|
"guessing" how long each will take, and for example you might schedule
|
|
them 5 minutes apart just to be safe, even if a given step only needs
|
|
2 minutes. (So another benefit really, is just avoiding the hassle of
|
|
scheduling all those steps separately to begin with.)
|
|
|
|
The other benefit is to allow for *restarting* the overnight tasks.
|
|
Luigi can keep track of which steps have ran thus far, so if there is
|
|
a failure, which you then address, you can restart the overnight task
|
|
and Luigi will pick up where it left off. (Steps which already ran
|
|
successfully will be skipped over.)
|