Install

AIQC Python Package

[ ]:

pip install --upgrade pip
pip install --upgrade wheel
pip install --upgrade aiqc

If during troubleshooting you find yourself reinstalling unwanted packages from the cache, then use:

pip install --upgrade --no-cache-dir aiqc

If that doesn’t work, read the rest of this notebook (e.g. supported Python versions).

Environment Setup

AIQC has many dependencies with specific versions, so we recommend creating a new virtual environment that is solely dedicated to AIQC using either PyEnv or Conda.

Python Version

Requires Python 3+ (check your deep learning library’s Python requirements). AIQC was developed on Python 3.7.12 in order to ensure compatibility with Google Colab.

Conda does not provide 3.7.12, but AIQC has been tested on 3.7.16 as well so you can use that version.

Additionally, check the Python version required by the machine learning libraries that you intend to use. For example, at the time this was written, Tensorflow/ Keras required Python 3.5–3.8. If you need more information about dependencies, the PyPI setup.py is in the root of the github.com/aiqc/aiqc repository.

[1]:

import sys
sys.version

[1]:

'3.7.12 (default, Dec 10 2021, 10:49:04) \n[Clang 13.0.0 (clang-1300.0.29.3)]'

Pickle Disclaimer

AIQC, much like PyTorch, relies heavily on Pickle for saving Python objects in its database. Therefore, as a caveat of Pickle, if you create objects in your aiqc.sqlite file using one version of Python and try to interact with it on a newer version of Python, then you may find that pickle is no longer able to deserialize the object. For this reason, sys.version and other helpful info about your OS/ Python version is stored in the config.json file at the time of creation.

Operating System

AIQC was designed to be OS-agnostic. It has been tested on the following operating systems:

macOS 10.15 and 11.6.1
Linux (Ubuntu, Alpine, RHEL).
Windows 10 (and WSL).

If you run into trouble with the installation process on your OS, please create a GitHub discussion so that we can attempt to resolve, document, and release a fix as quickly as possible.

Optional - JupyterLab IDE

AIQC runs anywhere Python runs. We just like Jupyter for interactive visualization and data transformation. FYI, jupyterlab is not an official dependency of the AIQC package.

[ ]:

pip install jupyterlab

JupyterLab requires Node.js >= 10. Once all extensions switch to JupyterLab 3.0 prebuilding, this will no longer be necessary.

[4]:

!node -v

v14.7.0

Optional - Swap Space for Failover Memory

On local machines, it is good practice to configure “swap space.” This way, if your processes run out of memory/ RAM, then the excess information will simply spill over onto the (potentially dynamically sized) swap partition of your hard drive, as opposed to causing an out-of-memory crash. For GB sized datasets, spinning media HDDs (5,400/ 7,200 RPM) may be too slow for usage with swap, but you could get by with NVMe/ SSD.

Location of AIQC Files

AIQC makes use of the Python package, appdirs, for an operating system (OS) agnostic location to store configuration and database files. This not only keeps your $HOME directory clean, but also helps prevent careless users from deleting your database.

The installation process checks not only that the corresponding appdirs folder exists on your system but also that you have the permissions neceessary to read from and write to that location. If these conditions are not met, then you will be provided instructions during the installation about how to create the folder and/ or grant yourself the appropriate permissions.

We have attempted to support both Windows (icacls permissions and backslashes C:\\) as well as POSIX including Mac and Linux including containers & Google Colab (chmod letters permissions and slashes /). Note: due to variations in the ordering of appdirs author and app directories in different OS’, we do not make use of the appdirs appauthor directory, only the appname directory.

Location Based on OS

Test it for yourself:

import appdirs; appdirs.user_data_dir('aiqc');

Mac: /Users/Username/Library/Application Support/aiqc
Linux - Alpine and Ubuntu: /root/.local/share/aiqc
Windows: C:\Users\Username\AppData\Local\aiqc

Database

The database is simply a SQLite file, and AIQC serves as an ORM/ API for that SQL database.

So you do not have to worry about anything like installing a database server, database client, database users, configuring ports, configuring passwords/ secrets/ environment variables, or starting and restopping the database. Shoutout to the ORM, peewee. Glad we found this fantastic and simple alternative to SQLAlchemy.

Config

The configuration file contains low level information about: * Where AIQC should persist data. * Runtime (Python, OS) environment for reproducibility and troubleshooting.

Optional - Deleting the Database

If, for whatever reason, you find that you need to destroy your SQLite database file and start from scratch, then you can do so without having to manually find and rm the database file. In order to reduce the chance of an accident, confirm:bool=False by default.

Bear in mind that if you are on either a server or shared OS, then this database may contain more than just your data.

a) One-Liner

Both confirm:bool=False and rebuild:bool=False, so it only does what you command it to do.

[ ]:

from aiqc.orm import create_db, destroy_db

[4]:

destroy_db(confirm=True, rebuild=True)


=> Success - deleted database file at path:
/Users/layne/Library/Application Support/aiqc/aiqc.sqlite3


=> Success - created database file at path:
/Users/layne/Library/Application Support/aiqc/aiqc.sqlite3


💾  Success - created all database tables.  💾

b) Or Line-by-Line

[5]:

destroy_db(confirm=True)


=> Success - deleted database file at path:
/Users/layne/Library/Application Support/aiqc/aiqc.sqlite3

[6]:

create_db()


=> Success - created database file at path:
/Users/layne/Library/Application Support/aiqc/aiqc.sqlite3


💾  Success - created all database tables.  💾

Troubleshooting

Reloading the Package

After CRUD’ing the config files, AIQC needs the be reimported in order to detect those changes. This can be done in one of three ways:

If everything goes smoothly, it should automatically happen behind the scenes: reload(sys.modules['aiqc']).
Manually by the user: from importlib import reload; reload(aiqc).
Manually restarting your Python kernel/ session and import aiqc.