Add some basic info on overnight luigi tasks
This commit is contained in:
parent
d32bfce883
commit
5aca67709e
|
@ -2,10 +2,67 @@
|
||||||
Scheduled Tasks
|
Scheduled Tasks
|
||||||
===============
|
===============
|
||||||
|
|
||||||
TODO
|
By "scheduled task" we mean, a command which runs automatically at a
|
||||||
|
pre-determined time. See also :doc:`commands` and :doc:`scripts`.
|
||||||
|
|
||||||
|
For now, Rattail assumes that any scheduled tasks will be manually
|
||||||
|
configured to run via `cron`_. See your operating system's
|
||||||
|
documentation for details.
|
||||||
|
|
||||||
|
.. _cron: https://en.wikipedia.org/wiki/Cron
|
||||||
|
|
||||||
|
(In the future, Rattail may add a `dedicated scheduler daemon`_, but
|
||||||
|
it is not yet a priority.)
|
||||||
|
|
||||||
|
.. _dedicated scheduler daemon: https://redmine.rattailproject.org/issues/13
|
||||||
|
|
||||||
|
|
||||||
Luigi
|
Luigi
|
||||||
-----
|
-----
|
||||||
|
|
||||||
TODO
|
`Luigi`_ is a "task runner" of sorts. Rattail uses it to help
|
||||||
|
orchestrate a "batch" of commands such as is often needed for
|
||||||
|
overnight automation (aka. "EOD").
|
||||||
|
|
||||||
|
.. _Luigi: https://luigi.readthedocs.io/en/stable/
|
||||||
|
|
||||||
|
The general idea is to maintain a Python module containing all of the
|
||||||
|
individual tasks which need to run. Rattail can then invoke the
|
||||||
|
module in such a way that Luigi runs all the tasks, keeping track of
|
||||||
|
which tasks have (not yet) ran, and in which order they should run
|
||||||
|
etc.
|
||||||
|
|
||||||
|
|
||||||
|
Overnight
|
||||||
|
~~~~~~~~~
|
||||||
|
|
||||||
|
Often there is just one "overnight" module, which contains all tasks.
|
||||||
|
But in some cases you may want a couple of separate modules, to run at
|
||||||
|
different times. A common example is like:
|
||||||
|
|
||||||
|
* run "all" (typical EOD) tasks, starting at 1am
|
||||||
|
* run "backups" for all servers, starting at 4am
|
||||||
|
|
||||||
|
In this example the "all" task may include a dozen or more individual
|
||||||
|
"steps" (which, confusingly, are also called "tasks"). Maybe data is
|
||||||
|
exported from one system to another, reports are ran, etc. Then the
|
||||||
|
"backups" task runs later, and may itself also contain multiple steps,
|
||||||
|
or perhaps just one.
|
||||||
|
|
||||||
|
There are two main benefits from wiring up overnight tasks via modules
|
||||||
|
to be ran by Luigi.
|
||||||
|
|
||||||
|
The first is that each individual step will (typically) start
|
||||||
|
immediately after the previous one completes. This means all steps
|
||||||
|
are "packed in" as tightly as possible, to help keep overall run time
|
||||||
|
down. Whereas scheduling each step separately often involves
|
||||||
|
"guessing" how long each will take, and for example you might schedule
|
||||||
|
them 5 minutes apart just to be safe, even if a given step only needs
|
||||||
|
2 minutes. (So another benefit really, is just avoiding the hassle of
|
||||||
|
scheduling all those steps separately to begin with.)
|
||||||
|
|
||||||
|
The other benefit is to allow for *restarting* the overnight tasks.
|
||||||
|
Luigi can keep track of which steps have ran thus far, so if there is
|
||||||
|
a failure, which you then address, you can restart the overnight task
|
||||||
|
and Luigi will pick up where it left off. (Steps which already ran
|
||||||
|
successfully will be skipped over.)
|
||||||
|
|
Loading…
Reference in a new issue