AIQC Python Package
pip install --upgrade pip pip install --upgrade wheel pip install --upgrade aiqc
If during troubleshooting you find yourself reinstalling unwanted packages from the cache, then use:
pip install --upgrade --no-cache-dir aiqc
If that doesn’t work, read the rest of this notebook (e.g. supported Python versions).
Requires Python 3+ (check your deep learning library’s Python requirements). AIQC was developed on Python 3.7.12 in order to ensure compatibility with Google Colab.
Additionally, check the Python version required by the machine learning libraries that you intend to use. For example, at the time this was written, Tensorflow/ Keras required Python 3.5–3.8. If you need more information about dependencies, the PyPI
setup.py is in the root of the github.com/aiqc/aiqc repository.
import sys sys.version
'3.7.12 (default, Dec 10 2021, 10:49:04) \n[Clang 13.0.0 (clang-1300.0.29.3)]'
AIQC, much like PyTorch, relies heavily on Pickle for saving Python objects in its database. Therefore, as a caveat of Pickle, if you create objects in your
aiqc.sqlite file using one version of Python and try to interact with it on a newer version of Python, then you may find that pickle is no longer able to deserialize the object. For this reason,
sys.version and other helpful info about your OS/ Python version is stored in the
config.json file at the time of creation.
AIQC was designed to be OS-agnostic. It has been tested on the following operating systems:
macOS 10.15 and 11.6.1
Linux (Ubuntu, Alpine, RHEL).
Windows 10 (and WSL).
If you run into trouble with the installation process on your OS, please create a GitHub discussion so that we can attempt to resolve, document, and release a fix as quickly as possible.
Optional - JupyterLab IDE
AIQC runs anywhere Python runs. We just like Jupyter for interactive visualization and data transformation. FYI, jupyterlab is not an official dependency of the AIQC package.
pip install jupyterlab
JupyterLab requires Node.js >= 10. Once all extensions switch to JupyterLab 3.0 prebuilding, this will no longer be necessary.
Optional - Swap Space for Failover Memory
On local machines, it is good practice to configure “swap space.” This way, if your processes run out of memory/ RAM, then the excess information will simply spill over onto the (potentially dynamically sized) swap partition of your hard drive, as opposed to causing an out-of-memory crash. For GB sized datasets, spinning media HDDs (5,400/ 7,200 RPM) may be too slow for usage with swap, but you could get by with NVMe/ SSD.
Location of AIQC Files
AIQC makes use of the Python package,
appdirs, for an operating system (OS) agnostic location to store configuration and database files. This not only keeps your
$HOME directory clean, but also helps prevent careless users from deleting your database.
The installation process checks not only that the corresponding appdirs folder exists on your system but also that you have the permissions neceessary to read from and write to that location. If these conditions are not met, then you will be provided instructions during the installation about how to create the folder and/ or grant yourself the appropriate permissions.
We have attempted to support both Windows (
icaclspermissions and backslashes
C:\\) as well as POSIX including Mac and Linux including containers & Google Colab (
chmod letterspermissions and slashes
/). Note: due to variations in the ordering of appdirs author and app directories in different OS’, we do not make use of the appdirs
appauthordirectory, only the
Location Based on OS
Test it for yourself:
import appdirs; appdirs.user_data_dir('aiqc');
Linux - Alpine and Ubuntu:
The database is simply a SQLite file, and AIQC serves as an ORM/ API for that SQL database.
So you *do not* have to worry about anything like installing a database server, database client, database users, configuring ports, configuring passwords/ secrets/ environment variables, or starting and restopping the database. Shoutout to the ORM, peewee. Glad we found this fantastic and simple alternative to SQLAlchemy.
The configuration file contains low level information about: * Where AIQC should persist data. * Runtime (Python, OS) environment for reproducibility and troubleshooting.
Optional - Deleting the Database
If, for whatever reason, you find that you need to destroy your SQLite database file and start from scratch, then you can do so without having to manually find and
rm the database file. In order to reduce the chance of an accident,
confirm:bool=False by default.
Bear in mind that if you are on either a server or shared OS, then this database may contain more than just your data.
rebuild:bool=False, so it only does what you command it to do.
from aiqc.orm import create_db, destroy_db
=> Success - deleted database file at path: /Users/layne/Library/Application Support/aiqc/aiqc.sqlite3 => Success - created database file at path: /Users/layne/Library/Application Support/aiqc/aiqc.sqlite3 💾 Success - created all database tables. 💾
b) Or Line-by-Line
=> Success - deleted database file at path: /Users/layne/Library/Application Support/aiqc/aiqc.sqlite3
=> Success - created database file at path: /Users/layne/Library/Application Support/aiqc/aiqc.sqlite3 💾 Success - created all database tables. 💾
Reloading the Package
After CRUD’ing the config files, AIQC needs the be reimported in order to detect those changes. This can be done in one of three ways:
If everything goes smoothly, it should automatically happen behind the scenes:
Manually by the user:
from importlib import reload; reload(aiqc).
Manually restarting your Python kernel/ session and