From 0726a70192b1af159b0a694c272b9505c734f8f2 Mon Sep 17 00:00:00 2001 From: Lance Edgar Date: Wed, 21 Jul 2021 18:15:02 -0500 Subject: [PATCH] Add some basic docs for data import/export --- docs/data/importing.rst | 5 ----- docs/data/importing/concepts.rst | 37 +++++++++++++++++++++++++++++++ docs/data/importing/exporting.rst | 33 +++++++++++++++++++++++++++ docs/data/importing/index.rst | 11 +++++++++ docs/data/importing/overview.rst | 25 +++++++++++++++++++++ docs/data/index.rst | 2 +- 6 files changed, 107 insertions(+), 6 deletions(-) delete mode 100644 docs/data/importing.rst create mode 100644 docs/data/importing/concepts.rst create mode 100644 docs/data/importing/exporting.rst create mode 100644 docs/data/importing/index.rst create mode 100644 docs/data/importing/overview.rst diff --git a/docs/data/importing.rst b/docs/data/importing.rst deleted file mode 100644 index 3afb840..0000000 --- a/docs/data/importing.rst +++ /dev/null @@ -1,5 +0,0 @@ - -Data Import / Export -==================== - -TODO diff --git a/docs/data/importing/concepts.rst b/docs/data/importing/concepts.rst new file mode 100644 index 0000000..f33a751 --- /dev/null +++ b/docs/data/importing/concepts.rst @@ -0,0 +1,37 @@ + +Concepts +======== + +.. _import-vs-export: + +Import vs. Export +----------------- + +From the framework's perspective, everything is actually an "import" +and never an "export" per se, i.e. data always flows from a source and +to a destination - or in other words, the "local" side always +*imports* from the "host" side. But see :doc:`exporting` for more +about that. + +.. _importer-vs-import-handler: + +Importer vs. Import Handler +--------------------------- + +All import (and export) tasks are accomplished by way of 2 things: + +The "importer" is responsible for reading data from one place and +writing it to another place. But it cannot run itself, it just +contains the e.g. table-specific read/write logic. + +The "import handler" is responsible for running one or more importers. +It also is (usually) responsible for "transaction-level" logic, e.g. +connecting to a database, and ultimately committing that transaction. +It usually has no table-specific logic, as that is the importer's job. + +For instance you might want to import all customers and products from +your POS into Rattail; this would require something like: + + * import handler for POS -> Rattail + * importer for Customer data + * importer for Product data diff --git a/docs/data/importing/exporting.rst b/docs/data/importing/exporting.rst new file mode 100644 index 0000000..0bc2066 --- /dev/null +++ b/docs/data/importing/exporting.rst @@ -0,0 +1,33 @@ + +Data Exporters +============== + +Within the framework there is no such thing as an "export" per se. +Everything functions as an import, i.e. the "local" side *imports* +from the "host" side. See :ref:`import-vs-export` for more on that. + +But there are a couple of caveats: + + +.. _declare-exporter: + +Declaring an Exporter +--------------------- + +In practice it's possible to mark any importer as being of the +"export" persuasion. This doesn't (usually) affect the underlying +logic, but is done e.g. to make command line usage and output more +sensible for us humans. + +To do this, set the +:attr:`~rattail:rattail.importing.handlers.ImportHandler.direction` +for your import handler to the value, ``'export'``. + + +.. _common-exporters: + +"Common" Exporters +------------------ + +TODO: describe e.g. SQLAlchemy -> CSV export, which dynamically +generates its own importer classes diff --git a/docs/data/importing/index.rst b/docs/data/importing/index.rst new file mode 100644 index 0000000..bb36553 --- /dev/null +++ b/docs/data/importing/index.rst @@ -0,0 +1,11 @@ + +Data Import / Export +==================== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + concepts + exporting diff --git a/docs/data/importing/overview.rst b/docs/data/importing/overview.rst new file mode 100644 index 0000000..81f50ec --- /dev/null +++ b/docs/data/importing/overview.rst @@ -0,0 +1,25 @@ + +Overview +======== + +Rattail offers a framework which may be used for (presumably) all data +import and export tasks, regardless of which systems are involved, or +direction of data flow between them. + +In other words it's not just for getting data into and out of a +Rattail DB, but e.g. might be used for exporting data from your POS to +an online mailing list, etc. + +See :doc:`concepts` for more background on the framework. + +The framework offers the following features: + + * "dry run" mode to check things out before committing changes + * "warnings" mode which sends email with data diffs, e.g. when you expect no changes + * adjustable "batch size" for grouping changes when submitting to local (target) system + * full command line support for above, plus "max" changes to apply, show progress, etc. + * core code is optimized to run quickly, e.g. by fetching all data up-front + * new importers may be created simply / cleanly / according to existing patterns + * new importers may extend / replace core functionality as needed + +See also the API docs for :mod:`rattail:rattail.importing` (and sub-modules). diff --git a/docs/data/index.rst b/docs/data/index.rst index fba96ad..c7cb74d 100644 --- a/docs/data/index.rst +++ b/docs/data/index.rst @@ -9,7 +9,7 @@ Data Layer db/index trainwreck other - importing + importing/index sync versioning batch/index