WuttaSync
=========

This provides a "batteries included" way to handle data sync between
arbitrary source and target.

This builds / depends on :doc:`WuttJamaican <wuttjamaican:index>`, for
sake of a common :term:`config object` and :term:`handler` interface.
It was originally designed for import to / export from the :term:`app
database` but **both** the source and target can be "anything" -
e.g. CSV or Excel file, cloud API, another DB.

The basic idea is as follows:

* read a data set from "source"
* read corresonding data from "target"
* compare the two data sets
* where they differ, create/update/delete records on the target

Although in some cases (e.g. export to CSV) the target has no
meaningful data so all source records are "created" on / written to
the target.

.. note::

   You may already have guessed, that this approach may not work for
   "big data" - and indeed, it is designed for "small" data sets,
   ideally 500K records or smaller.  It reads both (source/target)
   data sets into memory so that is the limiting factor.

   You can work around this to some extent, by limiting the data sets
   to a particular date range (or other "partitionable" aspect of the
   data), and only syncing that portion.

   However this is not meant to be an ETL engine involving a data
   lake/warehouse.  It is for more "practical" concerns where some
   disparate "systems" must be kept in sync, or basic import from /
   export to file.

The general "source → target" concept can be used for both import and
export, since "everything is an import" from the target's perspective.

In addition to the import/export framework proper, a CLI framework is
also provided.

A "real-time sync" framework is also (eventually) planned, similar to
the one developed in the Rattail Project;
cf. :doc:`rattail-manual:data/sync/index`.


.. toctree::
   :maxdepth: 2
   :caption: Documentation

   glossary
   narr/install
   narr/cli/index
   narr/concepts
   narr/custom/index

.. toctree::
   :maxdepth: 1
   :caption: API

   api/wuttasync
   api/wuttasync.cli
   api/wuttasync.cli.base
   api/wuttasync.cli.import_csv
   api/wuttasync.importing
   api/wuttasync.importing.base
   api/wuttasync.importing.csv
   api/wuttasync.importing.handlers
   api/wuttasync.importing.model
   api/wuttasync.importing.wutta
   api/wuttasync.util