wutta/wuttasync
3
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

feat: initial release

Browse source 
just the basics in place for an ad-hoc CSV -> Wutta import.  framework
API only at this point, no CLI yet
This commit is contained in:
Lance Edgar 2024-12-05 07:57:51 -06:00
commit e19eab418a
37 changed files with 3793 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.gitignore vendored Normal file
View file

@ -0,0 +1,4 @@
*~
*.pyc
.coverage
docs/_build/

12
CHANGELOG.md Normal file
View file

@ -0,0 +1,12 @@
# Changelog
All notable changes to WuttaSync will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
## v0.1.0 (??)
### Feat
- initial release

674
COPYING.txt Normal file
View file

@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.

6
README.md Normal file
View file

@ -0,0 +1,6 @@
# WuttaSync
Wutta framework for data import/export and real-time sync
See docs at https://rattailproject.org/docs/wuttasync/

20
docs/Makefile Normal file
View file

@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

6
docs/api/wuttasync.importing.base.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync.importing.base``
============================
.. automodule:: wuttasync.importing.base
:members:

6
docs/api/wuttasync.importing.csv.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync.importing.csv``
===========================
.. automodule:: wuttasync.importing.csv
:members:

6
docs/api/wuttasync.importing.handlers.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync.importing.handlers``
================================
.. automodule:: wuttasync.importing.handlers
:members:

6
docs/api/wuttasync.importing.model.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync.importing.model``
=============================
.. automodule:: wuttasync.importing.model
:members:

6
docs/api/wuttasync.importing.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync.importing``
=======================
.. automodule:: wuttasync.importing
:members:

6
docs/api/wuttasync.importing.wutta.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync.importing.wutta``
=============================
.. automodule:: wuttasync.importing.wutta
:members:

6
docs/api/wuttasync.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync``
=============
.. automodule:: wuttasync
:members:

6
docs/api/wuttasync.util.rst Normal file
View file

@ -0,0 +1,6 @@
``wuttasync.util``
==================
.. automodule:: wuttasync.util
:members:

40
docs/conf.py Normal file
View file

@ -0,0 +1,40 @@
# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
from importlib.metadata import version as get_version
project = 'WuttaSync'
copyright = '2024, Lance Edgar'
author = 'Lance Edgar'
release = get_version('WuttaSync')
# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
'sphinx.ext.viewcode',
'sphinx.ext.todo',
'enum_tools.autoenum',
]
templates_path = ['_templates']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
intersphinx_mapping = {
'python': ('https://docs.python.org/3/', None),
'wuttjamaican': ('https://rattailproject.org/docs/wuttjamaican/', None),
}
# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
html_theme = 'furo'
html_static_path = ['_static']

29
docs/index.rst Normal file
View file

@ -0,0 +1,29 @@
WuttaSync
=========
This package adds data import/export and real-time sync utilities for
the `Wutta Framework <https://wuttaproject.org>`_.
While it of course supports import/export to/from the Wutta :term:`app
database`, it may be used for any "source → target" data flow.
.. toctree::
:maxdepth: 2
:caption: Documentation
narr/install
.. toctree::
:maxdepth: 1
:caption: API
api/wuttasync
api/wuttasync.importing
api/wuttasync.importing.base
api/wuttasync.importing.csv
api/wuttasync.importing.handlers
api/wuttasync.importing.model
api/wuttasync.importing.wutta
api/wuttasync.util

35
docs/make.bat Normal file
View file

@ -0,0 +1,35 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
if "%1" == "" goto help
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd

12
docs/narr/install.rst Normal file
View file

@ -0,0 +1,12 @@
Installation
============
This assumes you already have a :doc:`WuttJamaican app+database
<wuttjamaican:narr/install/index>` setup and working.
Install the WuttaSync package to your virtual environment:
.. code-block:: sh
pip install WuttaSync

57
pyproject.toml Normal file
View file

@ -0,0 +1,57 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "WuttaSync"
version = "0.1.1"
description = "Wutta framework for data import/export and real-time sync"
readme = "README.md"
authors = [{name = "Lance Edgar", email = "lance@wuttaproject.org"}]
license = {text = "GNU GPL v3+"}
classifiers = [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
"Natural Language :: English",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
]
requires-python = ">= 3.8"
dependencies = [
"SQLAlchemy-Utils",
"WuttJamaican[db]",
]
[project.optional-dependencies]
docs = ["Sphinx", "enum-tools[sphinx]", "furo"]
tests = ["pytest-cov", "tox"]
[project.urls]
Homepage = "https://wuttaproject.org/"
Repository = "https://forgejo.wuttaproject.org/wutta/wuttasync"
Issues = "https://forgejo.wuttaproject.org/wutta/wuttasync/issues"
Changelog = "https://forgejo.wuttaproject.org/wutta/wuttasync/src/branch/master/CHANGELOG.md"
[tool.commitizen]
version_provider = "pep621"
tag_format = "v$version"
update_changelog_on_bump = true
[tool.hatch.build.targets.sdist]
exclude = [
"htmlcov/",
]
[tool.hatch.build.targets.wheel]
packages = ["src/wuttasync"]

27
src/wuttasync/__init__.py Normal file
View file

@ -0,0 +1,27 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
WuttaSync -- package root
"""
from ._version import __version__

6
src/wuttasync/_version.py Normal file
View file

@ -0,0 +1,6 @@
# -*- coding: utf-8; -*-
from importlib.metadata import version
__version__ = version('WuttaSync')

28
src/wuttasync/importing/__init__.py Normal file
View file

@ -0,0 +1,28 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
Data Import / Export Framework
"""
from .handlers import Orientation, ImportHandler, FromFileHandler, ToSqlalchemyHandler
from .base import Importer, FromFile, ToSqlalchemy

998
src/wuttasync/importing/base.py Normal file
View file

@ -0,0 +1,998 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
Data Importer base class
"""
import os
import logging
from sqlalchemy import orm
from sqlalchemy_utils.functions import get_primary_keys, get_columns
from wuttasync.util import data_diffs
log = logging.getLogger(__name__)
class Importer:
"""
Base class for all data importers / exporters.
So as with :class:`~wuttasync.importing.handlers.ImportHandler`,
despite the name ``Importer`` this class can be used for export as
well. Occasionally it's helpful to know "which mode" is in
effect, mostly for display to the user. See also
:attr:`orientation` and :attr:`actioning`.
The role of the "importer/exporter" (instance of this class) is to
process the import/export of data for **one "model"** - which
generally speaking, means one table. Whereas the "import/export
handler" (:class:`~wuttasync.importing.handlers.ImportHandler`
instance) orchestrates the overall DB connections, transactions
and invokes the importer(s)/exporter(s). So multiple
importers/exporters may run in the context of a single handler
job.
.. attribute:: handler
Reference to the parent
:class:`~wuttasync.importing.handlers.ImportHandler` instance.
.. attribute:: model_class
Reference to the :term:`data model` class representing the
target side, if applicable.
This normally would be a SQLAlchemy mapped class, e.g.
:class:`~wuttjamaican:wuttjamaican.db.model.base.Person` for
importing to the Wutta People table.
It is primarily (only?) used when the target side of the
import/export uses SQLAlchemy ORM.
"""
allow_create = True
"""
Flag indicating whether this importer/exporter should *ever* allow
records to be created on the target side.
This flag is typically defined in code for each handler.
See also :attr:`create`.
"""
allow_update = True
"""
Flag indicating whether this importer/exporter should *ever* allow
records to be updated on the target side.
This flag is typically defined in code for each handler.
See also :attr:`update`.
"""
allow_delete = True
"""
Flag indicating whether this importer/exporter should *ever* allow
records to be deleted on the target side.
This flag is typically defined in code for each handler.
See also :attr:`delete`.
"""
create = None
"""
Flag indicating the current import/export job should create
records on the target side, when applicable.
This flag is typically set by the caller, e.g. via command line
args.
See also :attr:`allow_create`.
"""
update = None
"""
Flag indicating the current import/export job should update
records on the target side, when applicable.
This flag is typically set by the caller, e.g. via command line
args.
See also :attr:`allow_update`.
"""
delete = None
"""
Flag indicating the current import/export job should delete
records on the target side, when applicable.
This flag is typically set by the caller, e.g. via command line
args.
See also :attr:`allow_delete`.
"""
caches_target = False
"""
Flag indicating the importer/exporter should pre-fetch the
existing target data. This is usually what we want, so both
source and target data sets are held in memory and lookups may be
done between them without additional fetching.
When this flag is false, the importer/exporter must query the
target for every record it gets from the source data, when looking
for a match.
"""
cached_target = False
"""
This is ``None`` unless :attr:`caches_target` is true, in which
case it may (at times) hold the result from
:meth:`get_target_cache()`.
"""
def __init__(self, config, **kwargs):
self.config = config
self.app = self.config.get_app()
self.create = kwargs.pop('create',
kwargs.pop('allow_create', self.allow_create))
self.update = kwargs.pop('update',
kwargs.pop('allow_update', self.allow_update))
self.delete = kwargs.pop('delete',
kwargs.pop('allow_delete', self.allow_delete))
self.__dict__.update(kwargs)
self.simple_fields = self.get_simple_fields()
self.supported_fields = self.get_supported_fields()
self.fields = self.get_fields()
@property
def orientation(self):
"""
Convenience property which returns the value of
:attr:`wuttasync.importing.handlers.ImportHandler.orientation`
from the parent import/export handler.
"""
return self.handler.orientation
@property
def actioning(self):
"""
Convenience property which returns the value of
:attr:`wuttasync.importing.handlers.ImportHandler.actioning`
from the parent import/export handler.
"""
return self.handler.actioning
@property
def dry_run(self):
"""
Convenience property which returns the value of
:attr:`wuttasync.importing.handlers.ImportHandler.dry_run`
from the parent import/export handler.
"""
return self.handler.dry_run
def get_model_title(self):
"""
Returns the display title for the target data model.
"""
if hasattr(self, 'model_title'):
return self.model_title
# TODO: this will fail if not using a model class, obviously..
# should raise more helpful error msg?
return self.model_class.__name__
def get_simple_fields(self):
"""
This should return a (possibly empty) list of "simple" fields
for the import/export. A "simple" field is one where the
value is a simple scalar, so e.g. can use ``getattr(obj,
field)`` to read and ``setattr(obj, field, value)`` to write.
See also :meth:`get_supported_fields()` and
:meth:`get_fields()`.
:returns: Possibly empty list of "simple" field names.
"""
if hasattr(self, 'simple_fields'):
return self.simple_fields
fields = get_columns(self.model_class)
return list(fields.keys())
def get_supported_fields(self):
"""
This should return the full list of fields which are available
for the import/export.
Note that this field list applies first and foremost to the
target side, i.e. if the target (table etc.) has no "foo"
field defined then it should not be listed here.
But it also applies to the source side, e.g. if target *does*
define a "foo" field but source does not, then it again should
not be listed here.
See also :meth:`get_simple_fields()` and :meth:`get_fields()`.
:returns: List of all "supported" field names.
"""
if hasattr(self, 'supported_fields'):
return self.supported_fields
return self.get_simple_fields()
def get_fields(self):
"""
This should return the "effective" list of fields which are to
be used for the import/export.
All fields in this list should also be found in the output for
:meth:`get_supported_fields()`.
See also :meth:`get_keys()` and :meth:`get_simple_fields()`.
:returns: List of "effective" field names.
"""
if hasattr(self, 'fields'):
return self.fields
return self.get_supported_fields()
def get_keys(self):
"""
Must return the key field(s) for use with import/export.
All fields in this list should also be found in the output for
:meth:`get_fields()`.
:returns: List of "key" field names.
"""
if hasattr(self, 'key'):
keys = self.key
if isinstance(keys, str):
keys = [keys]
return keys
return list(get_primary_keys(self.model_class))
def setup(self):
"""
Perform any setup needed before starting the import/export
job.
This is called from within :meth:`process_data()`. Default
logic does nothing.
"""
def teardown(self):
"""
Perform any teardown needed after ending the import/export
job.
This is called from within :meth:`process_data()`. Default
logic does nothing.
"""
def process_data(self, source_data=None, progress=None):
"""
Perform the data import/export operations on the target.
This is the core feature logic and may create, update and/or
delete records on the target side, depending on (subclass)
implementation. It is invoked directly by the parent
:attr:`handler`.
Note that subclass generally should not override this method,
but instead some of the others.
:param source_data: Optional sequence of normalized source
data. If not specified, it is obtained from
:meth:`normalize_source_data()`.
:param progress: Optional progress indicator factory.
:returns: A 3-tuple of ``(created, updated, deleted)`` as
follows:
* ``created`` - list of records created on the target
* ``updated`` - list of records updated on the target
* ``deleted`` - list of records deleted on the target
See also these methods which this one calls:
* :meth:`setup()`
* :meth:`do_create_update()`
* :meth:`do_delete()`
* :meth:`teardown()`
"""
self.setup()
created = []
updated = []
deleted = []
# get complete set of normalized source data
if source_data is None:
source_data = self.normalize_source_data(progress=progress)
# TODO: should exclude duplicate source records
# source_data, unique = self.get_unique_data(source_data)
# maybe cache existing target data
if self.caches_target:
self.cached_target = self.get_target_cache(source_data, progress=progress)
# create and/or update target data
if self.create or self.update:
created, updated = self.do_create_update(source_data, progress=progress)
# delete target data
if self.delete:
deleted = self.do_delete(source_data)
self.teardown()
return created, updated, deleted
def do_create_update(self, all_source_data, progress=None):
"""
Import/export the given normalized source data; create and/or
update target records as needed.
:param all_source_data: Sequence of all normalized source
data, e.g. as obtained from
:meth:`normalize_source_data()`.
:param progress: Optional progress indicator factory.
:returns: A 2-tuple of ``(created, updated)`` as follows:
* ``created`` - list of records created on the target
* ``updated`` - list of records updated on the target
This loops through all source data records, and for each will
try to find a matching target record. If a match is found it
also checks if any field values differ between them. So,
calls to these methods may also happen from here:
* :meth:`get_record_key()`
* :meth:`get_target_object()`
* :meth:`create_target_object()`
* :meth:`update_target_object()`
"""
model_title = self.get_model_title()
created = []
updated = []
# cache the set of fields to use for diff checks
fields = set(self.get_fields()) - set(self.get_keys())
def create_update(source_data, i):
# try to fetch target object per source key
key = self.get_record_key(source_data)
target_object = self.get_target_object(key)
if target_object and self.update:
# target object exists, so compare data
target_data = self.normalize_target_object(target_object)
diffs = self.data_diffs(source_data, target_data, fields=fields)
if diffs:
# data differs, so update target object
log.debug("fields (%s) differed for target data: %s and source data: %s",
','.join(diffs), target_data, source_data)
target_object = self.update_target_object(target_object,
source_data,
target_data=target_data)
updated.append((target_object, target_data, source_data))
elif not target_object and self.create:
# target object not yet present, so create it
target_object = self.create_target_object(key, source_data)
if target_object:
log.debug("created new %s %s: %s", model_title, key, target_object)
created.append((target_object, source_data))
# TODO: update cache if applicable
# if self.caches_target and self.cached_target is not None:
# self.cached_target[key] = {
# 'object': target_object,
# 'data': self.normalize_target_object(target_object),
# }
else:
log.debug("did NOT create new %s for key: %s", model_title, key)
actioning = self.actioning.capitalize()
target_title = self.handler.get_target_title()
self.app.progress_loop(create_update, all_source_data, progress,
message=f"{actioning} {model_title} data to {target_title}")
return created, updated
def do_delete(self, source_data, progress=None):
"""
TODO: not yet implemented
:returns: List of records deleted on the target.
"""
return []
def get_record_key(self, data):
"""
Returns the canonical key value for the given normalized data
record.
:param data: Normalized data record (dict).
:returns: A tuple of field values, corresponding to the
import/export key fields.
Note that this calls :meth:`get_keys()` to determine the
import/export key fields.
So if an importer has ``key = 'id'`` then :meth:`get_keys()`
would return ``('id',)`` and this method would return just the
``id`` value e.g. ``(42,)`` for the given data record.
The return value is always a tuple for consistency and to
allow for composite key fields.
"""
return tuple(data[key] for key in self.get_keys())
def data_diffs(self, source_data, target_data, fields=None):
"""
Find all (relevant) fields with differing values between the
two data records, source and target.
This is a simple wrapper around
:func:`wuttasync.util.data_diffs()` but unless caller
specifies a ``fields`` list, this will use the following by
default:
It calls :meth:`get_fields()` to get the effective field list,
and from that it *removes* the fields indicated by
:meth:`get_keys()`.
The thinking here, is that the goal of this function is to
find true diffs, but any "key" fields will already match (or
not) based on the overall processing logic and needn't be
checked further.
"""
if not fields:
# nb. we do not check for diffs on the key fields, since
# the source/target object matching already handles that
# effectively. also the uuid fields in particular can be
# tricky, if target schema uses UUID proper but source
# data represents them as string etc.
fields = set(self.get_fields()) - set(self.get_keys())
return data_diffs(source_data, target_data, fields=fields)
##############################
# source methods
##############################
def normalize_source_data(self, source_objects=None, progress=None):
"""
This method must return the full list of normalized data
records from the source.
Default logic here will call :meth:`get_source_objects()` and
then for each object :meth:`normalize_source_object_all()` is
called.
:param source_objects: Optional sequence of raw objects from
the data source. If not specified, it is obtained from
:meth:`get_source_objects()`.
:param progress: Optional progress indicator factory.
:returns: List of normalized source data records.
"""
if source_objects is None:
source_objects = self.get_source_objects()
normalized = []
def normalize(obj, i):
data = self.normalize_source_object_all(obj)
if data:
normalized.extend(data)
model_title = self.get_model_title()
source_title = self.handler.get_source_title()
self.app.progress_loop(normalize, source_objects, progress,
message=f"Reading {model_title} data from {source_title}")
return normalized
def get_source_objects(self):
"""
This method (if applicable) should return a sequence of "raw"
data objects (i.e. non-normalized records) from the source.
This method is typically called from
:meth:`normalize_source_data()` which then also handles the
normalization.
"""
return []
def normalize_source_object_all(self, obj):
"""
This method should "iterate" over the given object and return
a list of corresponding normalized data records.
In most cases, the object is "singular" and it doesn't really
make sense to return more than one data record for it. But
this method is here for subclass to override in those rare
cases where you *do* need to "expand" the object into multiple
source data records.
Default logic for this method simply calls
:meth:`normalize_source_object()` for the given object, and
returns a list with just that one record.
:param obj: Raw object from data source.
:returns: List of normalized data records corresponding to the
source object.
"""
data = self.normalize_source_object(obj)
if data:
return [data]
def normalize_source_object(self, obj):
"""
This should return a single "normalized" data record for the
given source object.
Subclass will usually need to override this, to "convert"
source data into the shared format required for import/export.
The default logic merely returns the object as-is!
Note that if this method returns ``None`` then the object is
effectively skipped, treated like it does not exist on the
source side.
:param obj: Raw object from data source.
:returns: Dict of normalized data fields, or ``None``.
"""
return obj
##############################
# target methods
##############################
def get_target_cache(self, source_data=None, progress=None):
"""
Fetch all (existing) raw objects and normalized data from the
target side, and return a cache object with all of that.
This method will call :meth:`get_target_objects()` first, and
pass along the ``source_data`` param if specified. From there
it will call :meth:`normalize_target_object()` and
:meth:`get_record_key()` for each.
:param source_data: Sequence of normalized source data for the
import/export job, if known.
:param progress: Optional progress indicator factory.
:returns: Dict whose keys are record keys (so one entry for
every normalized target record) and the values are a nested
dict with raw object and normalized record.
A minimal but complete example of what this return value
looks like::
{
(1,): {
'object': <some_object_1>,
'data': {'id': 1, 'description': 'foo'},
}
(2,): {
'object': <some_object_2>,
'data': {'id': 2, 'description': 'bar'},
}
}
"""
objects = self.get_target_objects(source_data=source_data)
cached = {}
def cache(obj, i):
data = self.normalize_target_object(obj)
if data:
key = self.get_record_key(data)
cached[key] = {'object': obj, 'data': data}
model_title = self.get_model_title()
target_title = self.handler.get_target_title()
self.app.progress_loop(cache, objects, progress,
message=f"Reading {model_title} data from {target_title}")
return cached
def get_target_objects(self, source_data=None, progress=None):
"""
Fetch all existing raw objects from the data target. Or at
least, enough of them to satisfy matching on the given source
data (if applicable).
:param source_data: Sequence of normalized source data for the
import/export job, if known.
:param progress: Optional progress indicator factory.
:returns: Corresponding sequence of raw objects from the
target side.
Note that the source data is provided only for cases where
that might be useful; it often is not.
But for instance if the source data contains say an ID field
and the min/max values present in the data set are 1 thru 100,
but the target side has millions of records, you might only
fetch ID <= 100 from target side as an optimization.
"""
raise NotImplementedError
def get_target_object(self, key):
"""
Should return the object from (existing) target data set which
corresponds to the given record key, if found.
Note that the default logic is able to find/return the object
from :attr:`cached_target` if applicable. But it is not able
to do a one-off lookup e.g. in the target DB. If you need the
latter then you should override this method.
:returns: Raw target data object, or ``None``.
"""
if self.caches_target and self.cached_target is not None:
cached = self.cached_target.get(key)
return cached['object'] if cached else None
def normalize_target_object(self, obj):
"""
This should return a "normalized" data record for the given
raw object from the target side.
Subclass will often need to override this, to "convert" target
object into the shared format required for import/export. The
default logic is only able to handle "simple" fields;
cf. :meth:`get_simple_fields()`.
It's possible to optimize this somewhat, by checking
:meth:`get_fields()` and then normalization may be skipped for
any fields which aren't "effective" for the current job.
Note that if this method returns ``None`` then the object is
ignored, treated like it does not exist on the target side.
:param obj: Raw object from data target.
:returns: Dict of normalized data fields, or ``None``.
"""
fields = self.get_fields()
fields = [f for f in self.get_simple_fields()
if f in fields]
data = dict([(field, getattr(obj, field))
for field in fields])
return data
##############################
# CRUD methods
##############################
def create_target_object(self, key, source_data):
"""
Create and return a new target object for the given key, fully
populated from the given source data. This may return
``None`` if no object is created.
This method will typically call :meth:`make_empty_object()`
and then :meth:`update_target_object()`.
:returns: New object for the target side, or ``None``.
"""
if source_data.get('__ignoreme__'):
return
obj = self.make_empty_object(key)
if obj:
return self.update_target_object(obj, source_data)
def make_empty_object(self, key):
"""
Return a new empty target object for the given key.
This method is called from :meth:`create_target_object()`. It
should only populate the object's key, and leave the rest of
the fields to :meth:`update_target_object()`.
Default logic will call :meth:`make_object()` to get the bare
instance, then populate just the fields from
:meth:`get_keys()`.
"""
obj = self.make_object()
for i, k in enumerate(self.get_keys()):
if hasattr(obj, k):
setattr(obj, k, key[i])
return obj
def make_object(self):
"""
Make a bare target object instance.
This method need not populate the object in any way. See also
:meth:`make_empty_object()`.
Default logic will make a new instance of :attr:`model_class`.
"""
return self.model_class()
def update_target_object(self, obj, source_data, target_data=None):
"""
Update the target object with the given source data, and
return the updated object.
This method may be called from :meth:`do_create_update()` for
a normal update, or :meth:`create_target_object()` when
creating a new record.
It should update the object for any of :meth:`get_fields()`
which appear to differ. However it need not bother for the
:meth:`get_keys()` fields, since those will already be
accurate.
:param obj: Raw target object.
:param source_data: Dict of normalized data for source record.
:param target_data: Dict of normalized data for existing
target record, if a typical update. Will be missing for a
new object.
:returns: The final updated object. In most/all cases this
will be the same instance as the original ``obj`` provided
by the caller.
"""
keys = self.get_keys()
fields = self.get_fields()
# we can automatically handle "simple" fields and update
# target object where needed for those
for field in self.get_simple_fields():
if field in keys:
# object key(s) should already be populated
continue
# elif field not in source_data:
# # no source data for field
# continue
elif field in fields:
# field is eligible for update generally, so compare
# values between records
if (not target_data
or field not in target_data
or target_data[field] != source_data[field]):
# data mismatch; update field for target object
setattr(obj, field, source_data[field])
return obj
class FromFile(Importer):
"""
Base class for importer/exporter using input file as data source.
Depending on the subclass, it may be able to "guess" (at least
partially) the path to the input file. If not, and/or to avoid
ambiguity, the caller must specify the file path.
In most cases caller may specify any of these via kwarg to the
class constructor, or e.g.
:meth:`~wuttasync.importing.handlers.ImportHandler.process_data()`:
* :attr:`input_file_path`
* :attr:`input_file_dir`
* :attr:`input_file_name`
The subclass itself can also specify via override of these
methods:
* :meth:`get_input_file_path()`
* :meth:`get_input_file_dir()`
* :meth:`get_input_file_name()`
And of course subclass must override these too:
* :meth:`open_input_file()`
* :meth:`close_input_file()`
* (and see also :attr:`input_file`)
.. attribute:: input_file_path
Path to the input file.
.. attribute:: input_file_dir
Path to folder containing input file(s).
.. attribute:: input_file_name
Name of the input file, sans folder path.
.. attribute:: input_file
Handle to the open input file, if applicable. May be set by
:meth:`open_input_file()` for later reference within
:meth:`close_input_file()`.
"""
def setup(self):
"""
Open the input file. See also :meth:`open_input_file()`.
"""
self.open_input_file()
def teardown(self):
"""
Close the input file. See also :meth:`close_input_file()`.
"""
self.close_input_file()
def get_input_file_path(self):
"""
This must return the full path to input file. It tries to
guess it based on various attributes, namely:
* :attr:`input_file_path`
* :attr:`input_file_dir`
* :attr:`input_file_name`
:returns: Path to input file.
"""
if hasattr(self, 'input_file_path'):
return self.input_file_path
folder = self.get_input_file_dir()
filename = self.get_input_file_name()
return os.path.join(folder, filename)
def get_input_file_dir(self):
"""
This must return the folder with input file(s). It tries to
guess it based on various attributes, namely:
* :attr:`input_file_dir`
:returns: Path to folder with input file(s).
"""
if hasattr(self, 'input_file_dir'):
return self.input_file_dir
raise NotImplementedError("can't guess path to input file(s) folder")
def get_input_file_name(self):
"""
This must return the input filename, sans folder path. It
tries to guess it based on various attributes, namely:
* :attr:`input_file_name`
:returns: Input filename, sans folder path.
"""
if hasattr(self, 'input_file_name'):
return self.input_file_name
raise NotImplementedError("can't guess input filename")
def open_input_file(self):
"""
Open the input file for reading source data.
Subclass must override to specify how this happens; default
logic is not implemented. Remember to set :attr:`input_file`
if applicable for reference when closing.
See also :attr:`get_input_file_path()` and
:meth:`close_input_file()`.
"""
raise NotImplementedError
def close_input_file(self):
"""
Close the input file for source data.
Subclass must override to specify how this happens; default
logic blindly calls the ``close()`` method on whatever
:attr:`input_file` happens to point to.
See also :attr:`open_input_file()`.
"""
self.input_file.close()
class ToSqlalchemy(Importer):
"""
Base class for importer/exporter which uses SQLAlchemy ORM on the
target side.
"""
caches_target = True
"" # nb. suppress sphinx docs
def get_target_object(self, key):
"""
Tries to fetch the object from target DB using ORM query.
"""
# first the default logic in case target object is cached
obj = super().get_target_object(key)
if obj:
return obj
# okay now we must fetch via query
query = self.target_session.query(self.model_class)
for i, k in enumerate(self.get_keys()):
query = query.filter(getattr(self.model_class, k) == key[i])
try:
return query.one()
except orm.exc.NoResultFound:
pass
def create_target_object(self, key, source_data):
""" """
with self.target_session.no_autoflush:
obj = super().create_target_object(key, source_data)
if obj:
# nb. add new object to target db session
self.target_session.add(obj)
return obj
def get_target_objects(self, source_data=None, progress=None):
"""
Fetches target objects via the ORM query from
:meth:`get_target_query()`.
"""
query = self.get_target_query(source_data=source_data)
return query.all()
def get_target_query(self, source_data=None):
"""
Returns an ORM query suitable to fetch existing objects from
the target side. This is called from
:meth:`get_target_objects()`.
"""
return self.target_session.query(self.model_class)

206
src/wuttasync/importing/csv.py Normal file
View file

@ -0,0 +1,206 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
Importing from CSV
"""
import csv
from collections import OrderedDict
from sqlalchemy_utils.functions import get_primary_keys
from wuttjamaican.db.util import make_topo_sortkey
from .base import FromFile
from .handlers import FromFileHandler
from .wutta import ToWuttaHandler
from .model import ToWutta
class FromCsv(FromFile):
"""
Base class for importer/exporter using CSV file as data source.
Note that this assumes a particular "format" for the CSV files.
If your needs deviate you should override more methods, e.g.
:meth:`open_input_file()`.
The default logic assumes CSV file is mostly "standard" - e.g.
comma-delimited, UTF-8-encoded etc. But it also assumes the first
line/row in the file contains column headers, and all subsequent
lines are data rows.
.. attribute:: input_reader
While the input file is open, this will reference a
:class:`python:csv.DictReader` instance.
"""
csv_encoding = 'utf_8'
"""
Encoding used by the CSV input file.
You can specify an override if needed when calling
:meth:`~wuttasync.importing.handlers.ImportHandler.process_data()`.
"""
def get_input_file_name(self):
"""
By default this returns the importer/exporter model name plus
CSV file extension, e.g. ``Widget.csv``
It calls
:meth:`~wuttasync.importing.base.Importer.get_model_title()`
to obtain the model name.
"""
if hasattr(self, 'input_file_name'):
return self.input_file_name
model_title = self.get_model_title()
return f'{model_title}.csv'
def open_input_file(self):
"""
Open the input file for reading, using a CSV parser.
This tracks the file handle via
:attr:`~wuttasync.importing.base.FromFile.input_file` and the
CSV reader via :attr:`input_reader`.
"""
path = self.get_input_file_path()
self.input_file = open(path, 'rt', encoding=self.csv_encoding)
self.input_reader = csv.DictReader(self.input_file)
def close_input_file(self):
""" """
self.input_file.close()
del self.input_reader
del self.input_file
def get_source_objects(self):
"""
This returns a list of data records "as-is" from the CSV
source file (via :attr:`input_reader`).
Since this uses :class:`python:csv.DictReader` by default,
each record will be a dict with key/value for each column in
the file.
"""
return list(self.input_reader)
class FromCsvToSqlalchemyMixin:
"""
Mixin handler class for CSV → SQLAlchemy ORM import/export.
"""
source_key = 'csv'
generic_source_title = "CSV"
FromImporterBase = FromCsv
"""
This must be set to a valid base class for the CSV source side.
Default is :class:`FromCsv` which should typically be fine; you
can change if needed.
"""
# nb. subclass must define this
ToImporterBase = None
"""
For a handler to use this mixin, this must be set to a valid base
class for the ORM target side. The :meth:`define_importers()`
logic will use this as base class when dynamically generating new
importer/exporter classes.
"""
def get_target_model(self):
"""
This should return the :term:`app model` or a similar module
containing data model classes for the target side.
The target model is used to dynamically generate a set of
importers (e.g. one per table in the target DB) which can use
CSV file as data source. See also :meth:`define_importers()`.
Subclass must override this if needed; default behavior is not
implemented.
"""
raise NotImplementedError
def define_importers(self):
"""
This mixin overrides typical (manual) importer definition, and
instead dynamically generates a set of importers, e.g. one per
table in the target DB.
It does this based on the target model, as returned by
:meth:`get_target_model()`. It calls
:meth:`make_importer_factory()` for each model class found.
"""
importers = {}
model = self.get_target_model()
# mostly try to make an importer for every data model
for name in dir(model):
cls = getattr(model, name)
if isinstance(cls, type) and issubclass(cls, model.Base) and cls is not model.Base:
importers[name] = self.make_importer_factory(cls, name)
# sort importers according to schema topography
topo_sortkey = make_topo_sortkey(model)
importers = OrderedDict([
(name, importers[name])
for name in sorted(importers, key=topo_sortkey)
])
return importers
def make_importer_factory(self, cls, name):
"""
Generate and return a new importer/exporter class, targeting
the given data model class.
:param cls: A data model class.
:param name: Optional "model name" override for the
importer/exporter.
:returns: A new class, meant to process import/export
operations which target the given data model. The new
class will inherit from both :attr:`FromImporterBase` and
:attr:`ToImporterBase`.
"""
return type(f'{name}Importer', (FromCsv, self.ToImporterBase), {
'model_class': cls,
'key': list(get_primary_keys(cls)),
})
class FromCsvToWutta(FromCsvToSqlalchemyMixin, ToWuttaHandler):
"""
Handler for CSV → Wutta :term:`app database` import.
"""
ToImporterBase = ToWutta
def get_target_model(self):
""" """
return self.app.model

560
src/wuttasync/importing/handlers.py Normal file
View file

@ -0,0 +1,560 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
Data Import / Export Handlers
"""
import logging
from collections import OrderedDict
from enum import Enum
from wuttjamaican.app import GenericHandler
log = logging.getLogger(__name__)
class Orientation(Enum):
"""
Enum values for :attr:`ImportHandler.orientation`.
"""
IMPORT = 'import'
EXPORT = 'export'
class ImportHandler(GenericHandler):
"""
Base class for all import/export handlers.
Despite the name ``ImportHandler`` this can be used for export as
well. The logic is no different on a technical level and the
"export" concept is mostly only helpful to the user. The latter
is important of course and to help with that we track the
:attr:`orientation` to distinguish.
The role of the "import/export handler" (instance of this class)
is to orchestrate the overall DB connections, transactions and
then invoke the importer/exporter instance(s) to do the actual
data assessment/transfer. Each of the latter will be an instance
of (a subclass of) :class:`~wuttasync.importing.base.Importer`.
"""
source_key = None
"""
Key identifier for the data source.
This should "uniquely" identify the data source, within the
context of the data target. For instance in the case of CSV →
Wutta, ``csv`` is the source key.
Among other things, this value is used in :meth:`get_key()`.
"""
target_key = None
"""
Key identifier for the data target.
This should "uniquely" identify the data target. For instance in
the case of CSV → Wutta, ``wutta`` is the target key.
Among other things, this value is used in :meth:`get_key()`.
"""
orientation = Orientation.IMPORT
"""
Orientation for the data flow. Must be a value from
:enum:`Orientation`:
* ``Orientation.IMPORT`` (aka. ``'import'``)
* ``Orientation.EXPORT`` (aka. ``'export'``)
Note that the value may be displayed to the user where helpful::
print(handler.orientation.value)
See also :attr:`actioning`.
It's important to understand the difference between import/export
and source/target; they are independent concepts. Source and
target indicate where data comes from and where it's going,
whereas import vs. export is mostly cosmetic.
How a given data flow's orientation is determined, is basically up
to the developer. Most of the time it is straightforward,
e.g. CSV → Wutta would be import, and Wutta → CSV would be
export. But confusing edge cases certainly exist, you'll know
them when you see them. In those cases the developer should try
to choose whichever the end user is likely to find less confusing.
"""
dry_run = False
"""
Flag indicating whether data import/export should truly happen vs.
dry-run only.
If true, the data transaction will be rolled back at the end; if
false then it will be committed.
See also :meth:`rollback_transaction()` and
:meth:`commit_transaction()`.
"""
importers = None
"""
This should be a dict of all importer/exporter classes available
to the handler. Keys are "model names" and each value is an
importer/exporter class. For instance::
{
'Widget': WidgetImporter,
}
This dict is defined during the handler constructor; see also
:meth:`define_importers()`.
Note that in practice, this is usually an ``OrderedDict`` so that
the "sorting" of importer/exporters can be curated.
If you want an importer/exporter instance you should not use this
directly but instead call :meth:`get_importer()`.
"""
def __init__(self, config, **kwargs):
""" """
super().__init__(config, **kwargs)
self.importers = self.define_importers()
def __str__(self):
""" """
return self.get_title()
@property
def actioning(self):
"""
Convenience property which effectively returns the
:attr:`orientation` in progressive verb tense - i.e. one of:
* ``'importing'``
* ``'exporting'``
"""
return f'{self.orientation.value}ing'
@classmethod
def get_key(cls):
"""
Returns the "full key" for the handler. This is a combination
of :attr:`source_key` and :attr:`target_key` and
:attr:`orientation`.
For instance in the case of CSV → Wutta, the full handler key
is ``to_wutta.from_csv.import``.
Note that more than one handler may return the same full key
here; but only one will be configured as the "default" handler
for that key. See also :meth:`get_spec()`.
"""
return f'to_{cls.target_key}.from_{cls.source_key}.{cls.orientation.value}'
@classmethod
def get_spec(cls):
"""
Returns the "class spec" for the handler. This value is the
same as what might be used to configure the default handler
for a given key.
For instance in the case of CSV → Wutta, the default handler
spec is ``wuttasync.importing.csv:FromCsvToWutta``.
See also :meth:`get_key()`.
"""
return f'{cls.__module__}:{cls.__name__}'
def get_title(self):
"""
Returns the full display title for the handler, e.g. ``"CSV →
Wutta"``.
Note that the :attr:`orientation` is not included in this title.
It calls :meth:`get_source_title()` and
:meth:`get_target_title()` to construct the full title.
"""
source = self.get_source_title()
target = self.get_target_title()
return f"{source} → {target}"
def get_source_title(self):
"""
Returns the display title for the data source.
See also :meth:`get_title()` and :meth:`get_target_title()`.
"""
if hasattr(self, 'source_title'):
return self.source_title
if hasattr(self, 'generic_source_title'):
return self.generic_source_title
return self.source_key
def get_target_title(self):
"""
Returns the display title for the data target.
See also :meth:`get_title()` and :meth:`get_source_title()`.
"""
if hasattr(self, 'target_title'):
return self.target_title
if hasattr(self, 'generic_target_title'):
return self.generic_target_title
return self.target_key
def process_data(self, *keys, **kwargs):
"""
Run import/export operations for the specified models.
:param \*keys: One or more importer/exporter (model) keys, as
defined by the handler.
Each key specified must be present in :attr:`importers` and
thus will correspond to an importer/exporter class.
A transaction is begun on the source and/or target side as
needed, then for each model key requested, the corresponding
importer/exporter is created and invoked. And finally the
transaction is committed (assuming normal operation).
See also these methods which may be called from this one:
* :meth:`consume_kwargs()`
* :meth:`begin_transaction()`
* :meth:`get_importer()`
* :meth:`~wuttasync.importing.base.Importer.process_data()` (on the importer/exporter)
* :meth:`rollback_transaction()`
* :meth:`commit_transaction()`
"""
kwargs = self.consume_kwargs(kwargs)
self.begin_transaction()
success = False
try:
# loop thru specified importer keys
for key in keys:
# invoke importer
importer = self.get_importer(key, **kwargs)
created, updated, deleted = importer.process_data()
# log what happened
msg = "%s: added %d; updated %d; deleted %d %s records"
if self.dry_run:
msg += " (dry run)"
log.info(msg, self.get_title(), len(created), len(updated), len(deleted), key)
except:
# TODO: what should happen here?
raise
else:
success = True
finally:
if not success:
log.warning("something failed, so transaction was rolled back")
self.rollback_transaction()
elif self.dry_run:
log.info("dry run, so transaction was rolled back")
self.rollback_transaction()
else:
log.info("transaction was committed")
self.commit_transaction()
def consume_kwargs(self, kwargs):
"""
This method is called by :meth:`process_data()`.
Its purpose is to give handlers a hook by which they can
update internal handler state from the given kwargs, prior to
running the import/export task(s).
Any kwargs which pertain only to the handler, should be
removed before they are returned. But any kwargs which (also)
may pertain to the importer/exporter instance, should *not* be
removed, so they are passed along via :meth:`get_importer()`.
:param kwargs: Dict of kwargs, "pre-consumption." This is the
same kwargs dict originally received by
:meth:`process_data()`.
:returns: Dict of kwargs, "post-consumption."
"""
if 'dry_run' in kwargs:
self.dry_run = kwargs['dry_run']
return kwargs
def begin_transaction(self):
"""
Begin an import/export transaction, on source and/or target
side as needed.
This is normally called from :meth:`process_data()`.
Default logic will call both:
* :meth:`begin_source_transaction()`
* :meth:`begin_target_transaction()`
"""
self.begin_source_transaction()
self.begin_target_transaction()
def begin_source_transaction(self):
"""
Begin a transaction on the source side, if applicable.
This is normally called from :meth:`begin_transaction()`.
"""
def begin_target_transaction(self):
"""
Begin a transaction on the target side, if applicable.
This is normally called from :meth:`begin_transaction()`.
"""
def commit_transaction(self):
"""
Commit the current import/export transaction, on source and/or
target side as needed.
This is normally called from :meth:`process_data()`.
Default logic will call both:
* :meth:`commit_target_transaction()`
* :meth:`commit_source_transaction()`
.. note::
By default the target transaction is committed first; this
is to avoid edge case errors when the source connection
times out. In such cases we want to properly cleanup the
target and then if an error happens when trying to cleanup
the source, it is less disruptive.
"""
# nb. it can sometimes be important to commit the target
# transaction first. in particular when the import takes a
# long time, it may be that no activity occurs on the source
# DB session after the initial data read. then at the end
# committing the source transaction may trigger a connection
# timeout error, which then prevents target transaction from
# committing. so now we just commit target first instead.
# TODO: maybe sequence should be configurable?
self.commit_target_transaction()
self.commit_source_transaction()
def commit_source_transaction(self):
"""
Commit the transaction on the source side, if applicable.
This is normally called from :meth:`commit_transaction()`.
"""
def commit_target_transaction(self):
"""
Commit the transaction on the target side, if applicable.
This is normally called from :meth:`commit_transaction()`.
"""
def rollback_transaction(self):
"""
Rollback the current import/export transaction, on source
and/or target side as needed.
This is normally called from :meth:`process_data()`. It is
"always" called when :attr:`dry_run` is true, but also may be
called if errors are encountered.
Default logic will call both:
* :meth:`rollback_target_transaction()`
* :meth:`rollback_source_transaction()`
.. note::
By default the target transaction is rolled back first;
this is to avoid edge case errors when the source
connection times out. In such cases we want to properly
cleanup the target and then if an error happens when trying
to cleanup the source, it is less disruptive.
"""
# nb. it can sometimes be important to rollback the target
# transaction first. in particular when the import takes a
# long time, it may be that no activity occurs on the source
# DB session after the initial data read. then at the end
# rolling back the source transaction may trigger a connection
# timeout error, which then prevents target transaction from
# being rolled back. so now we always rollback target first.
# TODO: maybe sequence should be configurable?
self.rollback_target_transaction()
self.rollback_source_transaction()
def rollback_source_transaction(self):
"""
Rollback the transaction on the source side, if applicable.
This is normally called from :meth:`rollback_transaction()`.
"""
def rollback_target_transaction(self):
"""
Rollback the transaction on the target side, if applicable.
This is normally called from :meth:`rollback_transaction()`.
"""
def define_importers(self):
"""
This method must "define" all importer/exporter classes
available to the handler. It is called from the constructor.
This should return a dict keyed by "model name" and each value
is an importer/exporter class. The end result is then
assigned as :attr:`importers` (in the constoructor).
For instance::
return {
'Widget': WidgetImporter,
}
Note that the model name will be displayed in various places
and the caller may invoke a specific importer/exporter by this
name etc. See also :meth:`get_importer()`.
"""
return OrderedDict()
def get_importer(self, key, **kwargs):
"""
Returns an importer/exporter instance corresponding to the
given key.
The key will be the "model name" mapped to a particular
importer/exporter class and thus must be present in
:attr:`importers`.
This method is called from :meth:`process_data()` but may also
be used by ad-hoc callers elsewhere.
It will call :meth:`get_importer_kwargs()` and then construct
the importer/exporter instance using those kwargs.
:param key: Model key for desired importer/exporter.
:returns: Instance of (subclass of)
:class:`~wuttasync.importing.base.Importer`.
"""
if key not in self.importers:
orientation = self.orientation.value
raise KeyError(f"unknown {orientation} key: {key}")
kwargs = self.get_importer_kwargs(key, **kwargs)
kwargs['handler'] = self
factory = self.importers[key]
return factory(self.config, **kwargs)
def get_importer_kwargs(self, key, **kwargs):
"""
Returns a dict of kwargs to be used when construcing an
importer/exporter with the given key. This is normally called
from :meth:`get_importer()`.
:param key: Model key for the desired importer/exporter,
e.g. ``'Widget'``
:param \**kwargs: Any kwargs we have so collected far.
:returns: Final kwargs dict for new importer/exporter.
"""
return kwargs
class FromFileHandler(ImportHandler):
"""
Handler for import/export which uses an input file as data source.
"""
class ToSqlalchemyHandler(ImportHandler):
"""
Handler for import/export which targets a SQLAlchemy ORM (DB).
"""
target_session = None
"""
Reference to the SQLAlchemy :term:`db session` for the target side.
This may often be a session for the :term:`app database` (i.e. for
importing to Wutta DB) but it could also be any other.
This will be ``None`` unless an import/export transaction is
underway. See also :meth:`begin_target_transaction()`.
"""
def begin_target_transaction(self):
"""
Establish a new :term:`db session` via
:meth:`make_target_session()` and assign the result to
:attr:`target_session`.
"""
self.target_session = self.make_target_session()
def rollback_target_transaction(self):
"""
Rollback the :attr:`target_session`.
"""
self.target_session.rollback()
self.target_session.close()
self.target_session = None
def commit_target_transaction(self):
"""
Commit the :attr:`target_session`.
"""
self.target_session.commit()
self.target_session.close()
self.target_session = None
def make_target_session(self):
"""
Make and return a new :term:`db session` for the import/export.
Subclass must override this; default logic is not implemented.
"""
raise NotImplementedError
def get_importer_kwargs(self, key, **kwargs):
""" """
kwargs = super().get_importer_kwargs(key, **kwargs)
kwargs.setdefault('target_session', self.target_session)
return kwargs

34
src/wuttasync/importing/model.py Normal file
View file

@ -0,0 +1,34 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
Data import/export to Wutta
"""
from .base import ToSqlalchemy
class ToWutta(ToSqlalchemy):
"""
Base class for data importer/exporter which targets the Wutta ORM
(:term:`app database`).
"""

57
src/wuttasync/importing/wutta.py Normal file
View file

@ -0,0 +1,57 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
Wutta ⇄ Wutta import/export
"""
from .handlers import ToSqlalchemyHandler
class ToWuttaHandler(ToSqlalchemyHandler):
"""
Handler for import/export which targets Wutta ORM (:term:`app
database`).
"""
target_key = 'wutta'
"" # nb. suppress docs
def get_target_title(self):
""" """
# nb. we override parent to use app title as default
if hasattr(self, 'target_title'):
return self.target_title
if hasattr(self, 'generic_target_title'):
return self.generic_target_title
return self.app.get_title()
def make_target_session(self):
"""
Call
:meth:`~wuttjamaican:wuttjamaican.app.AppHandler.make_session()`
and return the result.
:returns: :class:`~wuttjamaican:wuttjamaican.db.sess.Session`
instance.
"""
return self.app.make_session()

60
src/wuttasync/util.py Normal file
View file

@ -0,0 +1,60 @@
# -*- coding: utf-8; -*-
################################################################################
#
# WuttaSync -- Wutta framework for data import/export and real-time sync
# Copyright © 2024 Lance Edgar
#
# This file is part of Wutta Framework.
#
# Wutta Framework is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# Wutta Framework is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# Wutta Framework. If not, see <http://www.gnu.org/licenses/>.
#
################################################################################
"""
Data Utilities
"""
def data_diffs(source_data, target_data, fields=None):
"""
Find all (relevant) fields with differing values between the two
data records, source and target.
:param source_data: Dict of normalized record from source data.
:param target_data: Dict of normalized record from target data.
:param fields: Optional list of fields to check. If not
specified, all fields present in ``target_data`` will be
checked.
:returns: Possibly empty list of field names which were found to
differ between source and target record.
"""
if fields is None:
fields = list(target_data)
diffs = []
for field in fields:
if field not in target_data:
raise KeyError(f"field '{field}' is missing from target_data")
if field not in source_data:
raise KeyError(f"field '{field}' is missing from source_data")
target_value = target_data[field]
source_value = source_data[field]
if target_value != source_value:
diffs.append(field)
return diffs

24
tasks.py Normal file
View file

@ -0,0 +1,24 @@
# -*- coding: utf-8; -*-
"""
Tasks for WuttaSync
"""
import os
import shutil
from invoke import task
@task
def release(c, skip_tests=False):
"""
Release a new version of WuttaSync
"""
if not skip_tests:
c.run('pytest')
if os.path.exists('dist'):
shutil.rmtree('dist')
c.run('python -m build --sdist')
c.run('twine upload dist/*')

0
tests/__init__.py Normal file
View file

0
tests/importing/__init__.py Normal file
View file

437
tests/importing/test_base.py Normal file
View file

@ -0,0 +1,437 @@
#-*- coding: utf-8; -*-
from unittest.mock import patch
from wuttjamaican.testing import DataTestCase
from wuttasync.importing import base as mod, ImportHandler, Orientation
class TestImporter(DataTestCase):
def setUp(self):
self.setup_db()
self.handler = ImportHandler(self.config)
def make_importer(self, **kwargs):
kwargs.setdefault('handler', self.handler)
return mod.Importer(self.config, **kwargs)
def test_constructor(self):
model = self.app.model
# basic importer
imp = self.make_importer(model_class=model.Setting)
# fields
self.assertEqual(imp.supported_fields, ['name', 'value'])
self.assertEqual(imp.simple_fields, ['name', 'value'])
self.assertEqual(imp.fields, ['name', 'value'])
# orientation etc.
self.assertEqual(imp.orientation, Orientation.IMPORT)
self.assertEqual(imp.actioning, 'importing')
self.assertTrue(imp.create)
self.assertTrue(imp.update)
self.assertTrue(imp.delete)
self.assertFalse(imp.dry_run)
def test_get_model_title(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertEqual(imp.get_model_title(), 'Setting')
imp.model_title = "SeTtInG"
self.assertEqual(imp.get_model_title(), 'SeTtInG')
def test_get_simple_fields(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertEqual(imp.get_simple_fields(), ['name', 'value'])
imp.simple_fields = ['name']
self.assertEqual(imp.get_simple_fields(), ['name'])
def test_get_supported_fields(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertEqual(imp.get_supported_fields(), ['name', 'value'])
imp.supported_fields = ['name']
self.assertEqual(imp.get_supported_fields(), ['name'])
def test_get_fields(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertEqual(imp.get_fields(), ['name', 'value'])
imp.fields = ['name']
self.assertEqual(imp.get_fields(), ['name'])
def test_get_keys(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertEqual(imp.get_keys(), ['name'])
imp.key = 'value'
self.assertEqual(imp.get_keys(), ['value'])
def test_process_data(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting, caches_target=True)
# empty data set / just for coverage
with patch.object(imp, 'normalize_source_data') as normalize_source_data:
normalize_source_data.return_value = []
with patch.object(imp, 'get_target_cache') as get_target_cache:
get_target_cache.return_value = {}
result = imp.process_data()
self.assertEqual(result, ([], [], []))
def test_do_create_update(self):
model = self.app.model
# this requires a mock target cache
imp = self.make_importer(model_class=model.Setting, caches_target=True)
setting = model.Setting(name='foo', value='bar')
imp.cached_target = {
('foo',): {
'object': setting,
'data': {'name': 'foo', 'value': 'bar'},
},
}
# will update the one record
result = imp.do_create_update([{'name': 'foo', 'value': 'baz'}])
self.assertIs(result[1][0][0], setting)
self.assertEqual(result, ([], [(setting,
# nb. target
{'name': 'foo', 'value': 'bar'},
# nb. source
{'name': 'foo', 'value': 'baz'})]))
self.assertEqual(setting.value, 'baz')
# will create a new record
result = imp.do_create_update([{'name': 'blah', 'value': 'zay'}])
self.assertIsNot(result[0][0][0], setting)
setting_new = result[0][0][0]
self.assertEqual(result, ([(setting_new,
# nb. source
{'name': 'blah', 'value': 'zay'})],
[]))
self.assertEqual(setting_new.name, 'blah')
self.assertEqual(setting_new.value, 'zay')
# but what if new record is *not* created
with patch.object(imp, 'create_target_object', return_value=None):
result = imp.do_create_update([{'name': 'another', 'value': 'one'}])
self.assertEqual(result, ([], []))
# def test_do_delete(self):
# model = self.app.model
# imp = self.make_importer(model_class=model.Setting)
def test_get_record_key(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
record = {'name': 'foo', 'value': 'bar'}
self.assertEqual(imp.get_record_key(record), ('foo',))
imp.key = ('name', 'value')
self.assertEqual(imp.get_record_key(record), ('foo', 'bar'))
def test_data_diffs(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
# 2 identical records
rec1 = {'name': 'foo', 'value': 'bar'}
rec2 = {'name': 'foo', 'value': 'bar'}
result = imp.data_diffs(rec1, rec2)
self.assertEqual(result, [])
# now they're different
rec2['value'] = 'baz'
result = imp.data_diffs(rec1, rec2)
self.assertEqual(result, ['value'])
def test_normalize_source_data(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
# empty source data
data = imp.normalize_source_data()
self.assertEqual(data, [])
# now with 1 record
setting = model.Setting(name='foo', value='bar')
data = imp.normalize_source_data(source_objects=[setting])
self.assertEqual(len(data), 1)
# nb. default normalizer returns object as-is
self.assertIs(data[0], setting)
def test_get_source_objects(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertEqual(imp.get_source_objects(), [])
def test_normalize_source_object_all(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
setting = model.Setting()
result = imp.normalize_source_object_all(setting)
self.assertEqual(len(result), 1)
self.assertIs(result[0], setting)
def test_normalize_source_object(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
setting = model.Setting()
result = imp.normalize_source_object(setting)
self.assertIs(result, setting)
def test_get_target_cache(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
with patch.object(imp, 'get_target_objects') as get_target_objects:
get_target_objects.return_value = []
# empty cache
cache = imp.get_target_cache()
self.assertEqual(cache, {})
# cache w/ one record
setting = model.Setting(name='foo', value='bar')
get_target_objects.return_value = [setting]
cache = imp.get_target_cache()
self.assertEqual(len(cache), 1)
self.assertIn(('foo',), cache)
foo = cache[('foo',)]
self.assertEqual(len(foo), 2)
self.assertEqual(set(foo), {'object', 'data'})
self.assertIs(foo['object'], setting)
self.assertEqual(foo['data'], {'name': 'foo', 'value': 'bar'})
def test_get_target_objects(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertRaises(NotImplementedError, imp.get_target_objects)
def test_get_target_object(self):
model = self.app.model
setting = model.Setting(name='foo', value='bar')
# nb. must mock up a target cache for this one
imp = self.make_importer(model_class=model.Setting, caches_target=True)
imp.cached_target = {
('foo',): {
'object': setting,
'data': {'name': 'foo', 'value': 'bar'},
},
}
# returns same object
result = imp.get_target_object(('foo',))
self.assertIs(result, setting)
# and one more time just for kicks
result = imp.get_target_object(('foo',))
self.assertIs(result, setting)
# but then not if cache flag is off
imp.caches_target = False
result = imp.get_target_object(('foo',))
self.assertIsNone(result)
def test_normalize_target_object(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
setting = model.Setting(name='foo', value='bar')
data = imp.normalize_target_object(setting)
self.assertEqual(data, {'name': 'foo', 'value': 'bar'})
def test_create_target_object(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
# basic
setting = imp.create_target_object(('foo',), {'name': 'foo', 'value': 'bar'})
self.assertIsInstance(setting, model.Setting)
self.assertEqual(setting.name, 'foo')
self.assertEqual(setting.value, 'bar')
# will skip if magic delete flag is set
setting = imp.create_target_object(('foo',), {'name': 'foo', 'value': 'bar',
'__ignoreme__': True})
self.assertIsNone(setting)
def test_make_empty_object(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
obj = imp.make_empty_object(('foo',))
self.assertIsInstance(obj, model.Setting)
self.assertEqual(obj.name, 'foo')
def test_make_object(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
obj = imp.make_object()
self.assertIsInstance(obj, model.Setting)
def test_update_target_object(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
setting = model.Setting(name='foo')
# basic logic for updating *new* object
obj = imp.update_target_object(setting, {'name': 'foo', 'value': 'bar'})
self.assertIs(obj, setting)
self.assertEqual(setting.value, 'bar')
class TestFromFile(DataTestCase):
def setUp(self):
self.setup_db()
self.handler = ImportHandler(self.config)
def make_importer(self, **kwargs):
kwargs.setdefault('handler', self.handler)
return mod.FromFile(self.config, **kwargs)
def test_setup(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
with patch.object(imp, 'open_input_file') as open_input_file:
imp.setup()
open_input_file.assert_called_once_with()
def test_teardown(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
with patch.object(imp, 'close_input_file') as close_input_file:
imp.teardown()
close_input_file.assert_called_once_with()
def test_get_input_file_path(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
# path is guessed from dir+filename
path = self.write_file('data.txt', '')
imp.input_file_dir = self.tempdir
imp.input_file_name = 'data.txt'
self.assertEqual(imp.get_input_file_path(), path)
# path can be explicitly set
path2 = self.write_file('data2.txt', '')
imp.input_file_path = path2
self.assertEqual(imp.get_input_file_path(), path2)
def test_get_input_file_dir(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
# path cannot be guessed
self.assertRaises(NotImplementedError, imp.get_input_file_dir)
# path can be explicitly set
imp.input_file_dir = self.tempdir
self.assertEqual(imp.get_input_file_dir(), self.tempdir)
def test_get_input_file_name(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
# name cannot be guessed
self.assertRaises(NotImplementedError, imp.get_input_file_name)
# name can be explicitly set
imp.input_file_name = 'data.txt'
self.assertEqual(imp.get_input_file_name(), 'data.txt')
def test_open_input_file(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
self.assertRaises(NotImplementedError, imp.open_input_file)
def test_close_input_file(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
path = self.write_file('data.txt', '')
with open(path, 'rt') as f:
imp.input_file = f
with patch.object(f, 'close') as close:
imp.close_input_file()
close.assert_called_once_with()
class TestToSqlalchemy(DataTestCase):
def setUp(self):
self.setup_db()
self.handler = ImportHandler(self.config)
def make_importer(self, **kwargs):
kwargs.setdefault('handler', self.handler)
return mod.ToSqlalchemy(self.config, **kwargs)
def test_get_target_object(self):
model = self.app.model
setting = model.Setting(name='foo', value='bar')
# nb. must mock up a target cache for this one
imp = self.make_importer(model_class=model.Setting, caches_target=True)
imp.cached_target = {
('foo',): {
'object': setting,
'data': {'name': 'foo', 'value': 'bar'},
},
}
# returns same object
result = imp.get_target_object(('foo',))
self.assertIs(result, setting)
# and one more time just for kicks
result = imp.get_target_object(('foo',))
self.assertIs(result, setting)
# now let's put a 2nd setting in the db
setting2 = model.Setting(name='foo2', value='bar2')
self.session.add(setting2)
self.session.commit()
# then we should be able to fetch that via query
imp.target_session = self.session
result = imp.get_target_object(('foo2',))
self.assertIsInstance(result, model.Setting)
self.assertIs(result, setting2)
# but sometimes it will not be found
result = imp.get_target_object(('foo3',))
self.assertIsNone(result)
def test_create_target_object(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting, target_session=self.session)
setting = model.Setting(name='foo', value='bar')
# new object is added to session
setting = imp.create_target_object(('foo',), {'name': 'foo', 'value': 'bar'})
self.assertIsInstance(setting, model.Setting)
self.assertEqual(setting.name, 'foo')
self.assertEqual(setting.value, 'bar')
self.assertIn(setting, self.session)
def test_get_target_objects(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting, target_session=self.session)
setting1 = model.Setting(name='foo', value='bar')
self.session.add(setting1)
setting2 = model.Setting(name='foo2', value='bar2')
self.session.add(setting2)
self.session.commit()
result = imp.get_target_objects()
self.assertEqual(len(result), 2)
self.assertEqual(set(result), {setting1, setting2})

114
tests/importing/test_csv.py Normal file
View file

@ -0,0 +1,114 @@
#-*- coding: utf-8; -*-
import csv
from unittest.mock import patch
from wuttjamaican.testing import DataTestCase
from wuttasync.importing import csv as mod, ImportHandler, ToSqlalchemyHandler, ToSqlalchemy
class TestFromCsv(DataTestCase):
def setUp(self):
self.setup_db()
self.handler = ImportHandler(self.config)
def make_importer(self, **kwargs):
kwargs.setdefault('handler', self.handler)
return mod.FromCsv(self.config, **kwargs)
def test_get_input_file_name(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
# name can be guessed
self.assertEqual(imp.get_input_file_name(), 'Setting.csv')
# name can be explicitly set
imp.input_file_name = 'data.txt'
self.assertEqual(imp.get_input_file_name(), 'data.txt')
def test_open_input_file(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
path = self.write_file('data.txt', '')
imp.input_file_path = path
imp.open_input_file()
self.assertEqual(imp.input_file.name, path)
self.assertIsInstance(imp.input_reader, csv.DictReader)
imp.input_file.close()
def test_close_input_file(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
path = self.write_file('data.txt', '')
imp.input_file_path = path
imp.open_input_file()
imp.close_input_file()
self.assertFalse(hasattr(imp, 'input_reader'))
self.assertFalse(hasattr(imp, 'input_file'))
def test_get_source_objects(self):
model = self.app.model
imp = self.make_importer(model_class=model.Setting)
path = self.write_file('data.csv', """\
name,value
foo,bar
foo2,bar2
""")
imp.input_file_path = path
imp.open_input_file()
objects = imp.get_source_objects()
imp.close_input_file()
self.assertEqual(len(objects), 2)
self.assertEqual(objects[0], {'name': 'foo', 'value': 'bar'})
self.assertEqual(objects[1], {'name': 'foo2', 'value': 'bar2'})
class MockMixinHandler(mod.FromCsvToSqlalchemyMixin, ToSqlalchemyHandler):
ToImporterBase = ToSqlalchemy
class TestFromCsvToSqlalchemyMixin(DataTestCase):
def make_handler(self, **kwargs):
return MockMixinHandler(self.config, **kwargs)
def test_get_target_model(self):
with patch.object(mod.FromCsvToSqlalchemyMixin, 'define_importers', return_value={}):
handler = self.make_handler()
self.assertRaises(NotImplementedError, handler.get_target_model)
def test_define_importers(self):
model = self.app.model
with patch.object(mod.FromCsvToSqlalchemyMixin, 'get_target_model', return_value=model):
handler = self.make_handler()
importers = handler.define_importers()
self.assertIn('Setting', importers)
self.assertTrue(issubclass(importers['Setting'], mod.FromCsv))
self.assertTrue(issubclass(importers['Setting'], ToSqlalchemy))
self.assertIn('User', importers)
self.assertIn('Person', importers)
self.assertIn('Role', importers)
def test_make_importer_factory(self):
model = self.app.model
with patch.object(mod.FromCsvToSqlalchemyMixin, 'define_importers', return_value={}):
handler = self.make_handler()
factory = handler.make_importer_factory(model.Setting, 'Setting')
self.assertTrue(issubclass(factory, mod.FromCsv))
self.assertTrue(issubclass(factory, ToSqlalchemy))
class TestFromCsvToWutta(DataTestCase):
def make_handler(self, **kwargs):
return mod.FromCsvToWutta(self.config, **kwargs)
def test_get_target_model(self):
handler = self.make_handler()
self.assertIs(handler.get_target_model(), self.app.model)

218
tests/importing/test_handlers.py Normal file
View file

@ -0,0 +1,218 @@
#-*- coding: utf-8; -*-
from collections import OrderedDict
from unittest.mock import patch
from wuttjamaican.testing import DataTestCase
from wuttasync.importing import handlers as mod, Importer, ToSqlalchemy
class TestImportHandler(DataTestCase):
def make_handler(self, **kwargs):
return mod.ImportHandler(self.config, **kwargs)
def test_str(self):
handler = self.make_handler()
self.assertEqual(str(handler), "None → None")
handler.source_title = 'CSV'
handler.target_title = 'Wutta'
self.assertEqual(str(handler), "CSV → Wutta")
def test_actioning(self):
handler = self.make_handler()
self.assertEqual(handler.actioning, 'importing')
handler.orientation = mod.Orientation.EXPORT
self.assertEqual(handler.actioning, 'exporting')
def test_get_key(self):
handler = self.make_handler()
self.assertEqual(handler.get_key(), 'to_None.from_None.import')
with patch.multiple(mod.ImportHandler, source_key='csv', target_key='wutta'):
self.assertEqual(handler.get_key(), 'to_wutta.from_csv.import')
def test_get_spec(self):
handler = self.make_handler()
self.assertEqual(handler.get_spec(), 'wuttasync.importing.handlers:ImportHandler')
def test_get_title(self):
handler = self.make_handler()
self.assertEqual(handler.get_title(), "None → None")
handler.source_title = 'CSV'
handler.target_title = 'Wutta'
self.assertEqual(handler.get_title(), "CSV → Wutta")
def test_get_source_title(self):
handler = self.make_handler()
# null by default
self.assertIsNone(handler.get_source_title())
# which is really using source_key as fallback
handler.source_key = 'csv'
self.assertEqual(handler.get_source_title(), 'csv')
# can also use (defined) generic fallback
handler.generic_source_title = 'CSV'
self.assertEqual(handler.get_source_title(), 'CSV')
# or can set explicitly
handler.source_title = 'XXX'
self.assertEqual(handler.get_source_title(), 'XXX')
def test_get_target_title(self):
handler = self.make_handler()
# null by default
self.assertIsNone(handler.get_target_title())
# which is really using target_key as fallback
handler.target_key = 'wutta'
self.assertEqual(handler.get_target_title(), 'wutta')
# can also use (defined) generic fallback
handler.generic_target_title = 'Wutta'
self.assertEqual(handler.get_target_title(), 'Wutta')
# or can set explicitly
handler.target_title = 'XXX'
self.assertEqual(handler.get_target_title(), 'XXX')
def test_process_data(self):
model = self.app.model
handler = self.make_handler()
# empy/no-op should commit (not fail)
with patch.object(handler, 'commit_transaction') as commit_transaction:
handler.process_data()
commit_transaction.assert_called_once_with()
# do that again with no patch, just for kicks
handler.process_data()
# dry-run should rollback
with patch.object(handler, 'commit_transaction') as commit_transaction:
with patch.object(handler, 'rollback_transaction') as rollback_transaction:
handler.process_data(dry_run=True)
self.assertFalse(commit_transaction.called)
rollback_transaction.assert_called_once_with()
# and do that with no patch, for kicks
handler.process_data(dry_run=True)
# outright error should cause rollback
with patch.object(handler, 'commit_transaction') as commit_transaction:
with patch.object(handler, 'rollback_transaction') as rollback_transaction:
with patch.object(handler, 'get_importer', side_effect=RuntimeError):
self.assertRaises(RuntimeError, handler.process_data, 'BlahBlah')
self.assertFalse(commit_transaction.called)
rollback_transaction.assert_called_once_with()
# fake importer class/data
mock_source_objects = [{'name': 'foo', 'value': 'bar'}]
class SettingImporter(ToSqlalchemy):
model_class = model.Setting
target_session = self.session
def get_source_objects(self):
return mock_source_objects
# now for a "normal" one
handler.importers['Setting'] = SettingImporter
self.assertEqual(self.session.query(model.Setting).count(), 0)
handler.process_data('Setting')
self.assertEqual(self.session.query(model.Setting).count(), 1)
# then add another mock record
mock_source_objects.append({'name': 'foo2', 'value': 'bar2'})
handler.process_data('Setting')
self.assertEqual(self.session.query(model.Setting).count(), 2)
# nb. even if dry-run, record is added
# (rollback would happen later in that case)
mock_source_objects.append({'name': 'foo3', 'value': 'bar3'})
handler.process_data('Setting', dry_run=True)
self.assertEqual(self.session.query(model.Setting).count(), 3)
def test_consume_kwargs(self):
handler = self.make_handler()
# kwargs are returned as-is
kw = {}
result = handler.consume_kwargs(kw)
self.assertIs(result, kw)
# captures dry-run flag
self.assertFalse(handler.dry_run)
kw['dry_run'] = True
result = handler.consume_kwargs(kw)
self.assertIs(result, kw)
self.assertTrue(kw['dry_run'])
self.assertTrue(handler.dry_run)
def test_define_importers(self):
handler = self.make_handler()
importers = handler.define_importers()
self.assertEqual(importers, {})
self.assertIsInstance(importers, OrderedDict)
def test_get_importer(self):
model = self.app.model
handler = self.make_handler()
# normal
handler.importers['Setting'] = Importer
importer = handler.get_importer('Setting', model_class=model.Setting)
self.assertIsInstance(importer, Importer)
# key not found
self.assertRaises(KeyError, handler.get_importer, 'BunchOfNonsense', model_class=model.Setting)
class TestToSqlalchemyHandler(DataTestCase):
def make_handler(self, **kwargs):
return mod.ToSqlalchemyHandler(self.config, **kwargs)
def test_begin_target_transaction(self):
handler = self.make_handler()
with patch.object(handler, 'make_target_session') as make_target_session:
make_target_session.return_value = self.session
self.assertIsNone(handler.target_session)
handler.begin_target_transaction()
make_target_session.assert_called_once_with()
def test_rollback_target_transaction(self):
handler = self.make_handler()
with patch.object(handler, 'make_target_session') as make_target_session:
make_target_session.return_value = self.session
self.assertIsNone(handler.target_session)
handler.begin_target_transaction()
self.assertIs(handler.target_session, self.session)
handler.rollback_target_transaction()
self.assertIsNone(handler.target_session)
def test_commit_target_transaction(self):
handler = self.make_handler()
with patch.object(handler, 'make_target_session') as make_target_session:
make_target_session.return_value = self.session
self.assertIsNone(handler.target_session)
handler.begin_target_transaction()
self.assertIs(handler.target_session, self.session)
handler.commit_target_transaction()
self.assertIsNone(handler.target_session)
def test_make_target_session(self):
handler = self.make_handler()
self.assertRaises(NotImplementedError, handler.make_target_session)
def test_get_importer_kwargs(self):
handler = self.make_handler()
handler.target_session = self.session
kw = handler.get_importer_kwargs('Setting')
self.assertIn('target_session', kw)
self.assertIs(kw['target_session'], self.session)

3
tests/importing/test_model.py Normal file
View file

@ -0,0 +1,3 @@
#-*- coding: utf-8; -*-
from wuttasync.importing import model as mod

38
tests/importing/test_wutta.py Normal file
View file

@ -0,0 +1,38 @@
#-*- coding: utf-8; -*-
from unittest.mock import patch
from wuttjamaican.testing import DataTestCase
from wuttasync.importing import wutta as mod
class TestToWuttaHandler(DataTestCase):
def make_handler(self, **kwargs):
return mod.ToWuttaHandler(self.config, **kwargs)
def test_get_target_title(self):
handler = self.make_handler()
# uses app title by default
self.config.setdefault('wutta.app_title', "What About This")
self.assertEqual(handler.get_target_title(), 'What About This')
# or generic default if present
handler.generic_target_title = "WHATABOUTTHIS"
self.assertEqual(handler.get_target_title(), 'WHATABOUTTHIS')
# but prefer specific title if present
handler.target_title = "what_about_this"
self.assertEqual(handler.get_target_title(), 'what_about_this')
def test_make_target_session(self):
handler = self.make_handler()
# makes "new" (mocked in our case) app session
with patch.object(self.app, 'make_session') as make_session:
make_session.return_value = self.session
session = handler.make_target_session()
make_session.assert_called_once_with()
self.assertIs(session, self.session)

29
tests/test_util.py Normal file
View file

@ -0,0 +1,29 @@
# -*- coding: utf-8; -*-
from unittest import TestCase
from wuttasync import util as mod
class TestDataDiffs(TestCase):
def test_source_missing_field(self):
source = {'foo': 'bar'}
target = {'baz': 'xyz', 'foo': 'bar'}
self.assertRaises(KeyError, mod.data_diffs, source, target)
def test_target_missing_field(self):
source = {'foo': 'bar', 'baz': 'xyz'}
target = {'baz': 'xyz'}
self.assertRaises(KeyError, mod.data_diffs, source, target, fields=['foo', 'baz'])
def test_no_diffs(self):
source = {'foo': 'bar', 'baz': 'xyz'}
target = {'baz': 'xyz', 'foo': 'bar'}
self.assertFalse(mod.data_diffs(source, target))
def test_with_diffs(self):
source = {'foo': 'bar', 'baz': 'xyz'}
target = {'baz': 'xyz', 'foo': 'BAR'}
result = mod.data_diffs(source, target)
self.assertEqual(result, ['foo'])

17
tox.ini Normal file
View file

@ -0,0 +1,17 @@
[tox]
envlist = py38, py39, py310, py311
[testenv]
extras = tests
commands = pytest {posargs}
[testenv:coverage]
basepython = python3.11
commands = pytest --cov=wuttasync --cov-report=html --cov-fail-under=100
[testenv:docs]
basepython = python3.11
extras = docs
changedir = docs
commands = sphinx-build -b html -d {envtmpdir}/doctrees -W -T . {envtmpdir}/docs