Add most of the structure! plus several Base Layer docs

docs/base/config/db.rst

@@ -0,0 +1,86 @@
+
+.. highlight:: ini
+
+Storing Config in DB
+====================
+
+We're getting ahead of ourselves a little here, if you're reading this manual
+straight through.  But for reference sake this probably belongs here.
+
+Settings Table
+--------------
+
+If you already have a Rattail DB, then it has a table named ``setting`` which
+is designed to store config values.  It's just a regular table so you can write
+settings via SQL if you like:
+
+.. code-block:: sql
+
+   insert into setting (name, value) values ('rattail.app_title', 'Poser');
+
+   update setting set value = 'Something Else' where name = 'rattail.app_title';
+
+Although the main reason for putting settings in the DB is usually so you can
+edit them via the web app.
+
+
+Telling App to Read Config from DB
+----------------------------------
+
+Well first of all we must assume that the DB connection itself is configured.
+More on that later but let's say you have this in place::
+
+   [rattail.db]
+   default.url = postgresql://user:password@localhost/poser
+
+Then you also must tell the Rattail config engine that a) it should read config
+values from the DB at all, but probably also b) it should *prefer* values from
+the DB over what was read from file.  Do this within your config file::
+
+   [rattail.config]
+   usedb = true
+   preferdb = true
+
+With this in place, when the app requests a config value, it will come from the
+DB if present, falling back to the file value if that exists.
+
+
+File vs. DB Setting Names
+-------------------------
+
+You may have noticed that the SQL examples above, and the examples used in
+:doc:`syntax` are really for the same 'app_title' setting, but there is a key
+difference in naming.
+
+So in a config file you might have this snippet to define a setting::
+
+   [rattail]
+   app_title = Poser
+
+But then that same setting is written in SQL as:
+
+.. code-block:: sql
+
+   insert into setting (name, value) values ('rattail.app_title', 'Poser');
+
+In other words the "section" and "option" names from the config file, are
+joined together with a dot, for the DB setting name.
+
+When the app requests a config value, it must specify both a section and
+option.  The config engine then will auto-join them as needed when doing DB
+lookups.
+
+.. code-block:: python
+
+   config.get('rattail', 'app_title')
+
+
+Avoiding the DB
+---------------
+
+The app can request config from "file only" if it needs to.  It just has to
+specify a flag when reading the value, for example:
+
+.. code-block:: python
+
+   config.get('rattail', 'app_title', usedb=False)

docs/base/config/defined.rst

@@ -0,0 +1,11 @@
+
+Defined Settings
+================
+
+There are a number of config settings which a given Poser/Rattail app may
+leverage.  Most if not all are optional.  We'll try to describe them here.
+
+Note that we will present each in terms of a "section" and "option" pair,
+i.e. using the naming convention found in a config file vs. that of DB.
+
+TODO!

docs/base/config/generate.rst

@@ -0,0 +1,21 @@
+
+Generating Config Files
+=======================
+
+Rattail is able to generate "starting point" config files for you, for any of
+the types described in :doc:`paths`.  Usually you will need to edit them
+further, but the basic structure can be provided automatically.
+
+Run any of these commands from your env root (e.g. ``/srv/envs/poser``)
+depending on which types of files you need:
+
+.. code-block:: sh
+
+   bin/rattail make-config -O app/ -T rattail
+   bin/rattail make-config -O app/ -T quiet
+   bin/rattail make-config -O app/ -T web
+   bin/rattail make-config -O app/ -T datasync
+   bin/rattail make-config -O app/ -T filemon
+
+Each generated file should contain "TODO" comments directing your attention to
+settings which may require adjustment.

docs/base/config/index.rst

@@ -0,0 +1,16 @@
+
+Configuration
+=============
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   overview
+   syntax
+   paths
+   inheritance
+   generate
+   logging
+   db
+   defined

docs/base/config/inheritance.rst

@@ -0,0 +1,72 @@
+
+.. highlight:: ini
+
+Config File Inheritance
+=======================
+
+It may already be obvious, if you read :doc:`paths`, but it's possible for
+config files to "inherit" from one another.  The general idea is that all
+config is collected from various files, when assembling the "final" config to
+be used by the app.
+
+For a simple example let's assume you have just 2 typical config files in your
+app dir:
+
+* ``/srv/envs/poser/app/rattail.conf``
+* ``/srv/envs/poser/app/quiet.conf``
+
+Let's say that ``rattail.conf`` is a "complete" config file and may be used
+directly, as-is.  And that ``quiet.conf`` is not complete but "inherits" from
+``rattail.conf`` (as is typical) so that it also may be used directly.
+
+In other words either of these commands should work when ran from
+``/srv/envs/poser``:
+
+.. code-block:: sh
+
+   bin/rattail -c app/rattail.conf make-uuid
+   bin/rattail -c app/quiet.conf make-uuid
+
+The contents of ``quiet.conf`` are usually quite minimal::
+
+   [rattail.config]
+   include = %(here)s/rattail.conf
+
+   [handler_console]
+   level = INFO
+
+The "include" option within "rattail.config" section above, tells the Rattail
+config parser to bring in the contents of ``rattail.conf`` whenever it is
+reading ``quiet.conf`` - although any settings defined in ``quiet.conf`` will
+override whatever was brought in from ``rattail.conf``.  (In this example,
+``quiet.conf`` only needs to set ``level = INFO`` to cut down on some logging
+output on the console.)
+
+Caveats
+-------
+
+There is a gotcha which can break the inheritance logic, but it can be avoided
+if you follow one simple rule:
+
+The primary config file you reference when invoking the app
+(e.g. ``bin/rattail -c qpp/quiet.conf ...``) must *not* contain a 'loggers'
+section, i.e. it should *not* have a snippet like this::
+
+   [loggers]
+   keys = root, exc_logger, ...
+
+To be clear the gotcha only exists when:
+
+* config file which app is told to read, contains snippet like above
+* config file has an ``include`` setting, meaning inheritance should happen
+* config file also says to configure logging
+
+The reason it breaks is that we let Python standard ``logging`` module take
+care of the logging configuration, but it will try to do so using the specified
+config file (e.g. ``quiet.conf``) *only* instead of doing so with the combined
+result.
+
+So again, just make sure there is no 'loggers' section in the config file you
+present to your app.  Or alternatively, you can make sure that same config file
+*does* have all logging config within it, so e.g. inheritance would not affect
+that part.

docs/base/config/logging.rst

@@ -0,0 +1,32 @@
+
+.. highlight:: ini
+
+Configuring Logging
+===================
+
+Rattail relies on Python standard ``logging`` module to configure logging.
+However if :doc:`inheritance` is involved then Rattail will first combine all
+applicable config files into a single file before handing that off to
+`logging.config.fileConfig()`_.
+
+.. _logging.config.fileConfig(): https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig
+
+Rattail does not do any of this though, unless config says to.  So if you want
+it to configure logging in this way, specify this in your config file::
+
+   [rattail.config]
+   configure_logging = true
+
+Beyond that you must ensure your config file(s) contains appropriate settings
+for logging.  Rattail `has a sample`_ ``rattail.conf`` which includes a typical
+logging section.  (This is the source used to generate a new file when you run
+``rattail make-config -T rattail`` command.)
+
+.. _has a sample: https://kallithea.rattailproject.org/rattail-project/rattail/files/master/rattail/data/config/rattail.conf
+
+See the Python docs for more info, in particular these sections:
+
+* `Configuration file format <https://docs.python.org/3/library/logging.config.html#logging-config-fileformat>`_
+* `Logging Levels <https://docs.python.org/3/howto/logging.html#logging-levels>`_
+* `Useful Handlers <https://docs.python.org/3/howto/logging.html#useful-handlers>`_
+* `Formatters <https://docs.python.org/3/howto/logging.html#formatters>`_

docs/base/config/overview.rst

@@ -0,0 +1,31 @@
+
+Overview
+========
+
+The basic idea of course is that the app needs to be configurable, so the
+question is how to go about that.
+
+The short answer is, "We always use config files but can also store config in
+the database where applicable."
+
+The advantage to config files is that they are easier to get started with, but
+also we can restrict access at the file system level, which means we can (more
+safely) store sensitive information in them.
+
+The advantage to storing config in DB, is that one can change config on-the-fly
+using e.g. the web app.  (Whereas changing a config file requires the app to be
+restarted, so it will read the new file contents.)
+
+Generally speaking, certain config which is needed during app startup (e.g. DB
+access credentials and logging config), or which is deemed "sensitive"
+(passwords, API keys etc.), is kept in config files, and the remainder is kept
+in the DB (if there is one).
+
+This behavior is itself configurable, but most typically, the app will not care
+where a given setting is defined (file vs. DB) but if one is defined in both
+places the DB value would win.  But the app can request a config value from
+"file only" and ignore the DB, where that is desirable (e.g. when reading
+settings required for startup).
+
+We are still in the Base Layer docs at this point, so we'll focus on the config
+files first, and come back to the DB near the end.

docs/base/config/paths.rst

@@ -0,0 +1,67 @@
+
+Typical File Paths
+==================
+
+Here we'll describe some typical locations and filenames for config.
+
+
+App-Wide
+--------
+
+Every app will need at least one config file.  The convention is to name this
+file ``rattail.conf`` and place it directly in your app dir, e.g.
+``/srv/envs/poser/app/rattail.conf``
+
+Many apps will benefit from having multiple config files, primarily for the
+sake of organization and "separation of concerns".  A robust app with several
+features then might have the following, all in its app dir:
+
+``rattail.conf`` is considered the "core" config file for the app, which means
+that no matter how you run the app, this file should be (in)directly referenced
+somehow, so that it affects the runtime behavior.
+
+``quiet.conf`` is meant to be used for ad-hoc app commands which you run from
+the console.  It is a thin wrapper around ``rattail.conf`` and merely tries to
+cut down on some of the output (logging) "noise" from commands.
+
+``cron.conf`` is also a thin wrapper, which cuts down on command output "noise"
+and uses a custom logging file.
+
+``web.conf`` is meant to be used by the standard web app only; it defines the
+config needed to run the web app and uses a custom logging file.
+
+``webapi.conf`` is meant to be used by the web API only; it defines the config
+needed to run the web API and uses a custom logging file.
+
+``datasync.conf`` is meant to be used only by the ``datasync`` commands; it
+defines the config needed to run datasync and uses a custom logging file.
+
+``filemon.conf`` is meant to be used only by the ``filemon`` commands; it
+defines the config needed to run filemon and uses a custom logging file.
+
+``bouncer.conf`` is meant to be used only by the ``bouncer`` commands; it
+defines the config needed to run bouncer and uses a custom logging file.
+
+
+Machine-Wide
+------------
+
+If you have several apps running on a given machine, you may wish to share some
+"common" config among all the apps.  If so you can make a "machine-wide" config
+file to store that common config.
+
+The convention here is to place the file at ``/etc/rattail/rattail.conf``
+although you could do whatever you like.
+
+
+Site-Wide
+---------
+
+This is similar to the machine-wide scenario, but if you have multiple
+*machines* which run apps, you may wish to share the common config in such a
+way that all machines could access it.  We call this a "site-wide" config file.
+
+How exactly you go about making this file available is beyond our scope here,
+but a hint is to use Samba/CIFS.  True location of the file would be anywhere
+you like, on whichever machine you elected to be the provider of this common
+config.

docs/base/config/syntax.rst

@@ -0,0 +1,59 @@
+
+.. highlight:: ini
+
+Config File Syntax
+==================
+
+Rattail config files follow the traditional `INI file`_ syntax, e.g. here is a
+snippet::
+
+   [rattail]
+   app_title = Poser
+
+.. _INI file: https://en.wikipedia.org/wiki/INI_file
+
+That example can be broken down into 3 parts:
+
+* section (the name in square brackets, "rattail")
+* option (the name of the setting being defined, "app_title")
+* value (the value for the setting, "Poser")
+
+When logic within the app needs to retrieve a value from config, it does so by
+requesting both the section and option by name, for example:
+
+.. code-block:: python
+
+   config.get('rattail', 'app_title')   # returns "Poser"
+
+Under the hood we use Python's `configparser`_ module to parse config files.
+
+.. _configparser: https://docs.python.org/3/library/configparser.html
+
+If you think it would be handy for your app to have a new setting for some
+reason, just create one and use it like so::
+
+   [poser]
+   foo = bar
+
+.. code-block:: python
+
+   config.get('poser', 'foo')   # returns "bar"
+
+So far our examples have been for simple string values.  There is no way within
+the standard INI file syntax, to define any data types other than string.
+However the Rattail config parser/object can "coerce" values to a given type if
+so requested, for example::
+
+   [poser]
+   foo_flag = true
+   foo_number = 42
+   foo_entries =
+       first
+       second
+       third
+
+.. code-block:: python
+
+   config.getbool('poser', 'foo_flag')      # returns True
+   config.getint('poser', 'foo_number')     # returns 42
+   config.getlist('poser', 'foo_entries')   # returns ["first", "second", "third"]