Getting Started

Installation

Install the latest release from PyPI:

python -m pip install cvi

To install the current development version directly from GitHub:

python -m pip install git+https://github.com/AP6YC/cvi.git

Batch evaluation

A batch is a two-dimensional NumPy array with one sample per row and one feature per column. Labels are a one-dimensional array with one integer label per sample.

import numpy as np
import cvi

samples = np.array([
    [0.0, 0.1],
    [0.2, 0.0],
    [2.8, 3.0],
    [3.1, 2.9],
])
labels = np.array([0, 1, 2, 2])

index = cvi.CH()
value = index.get_cvi(samples, labels)

The call updates index in place and returns the resulting criterion value. A CVI object accepts only one batch initialization. Create a new object to evaluate an independent partition.

Streaming evaluation

Pass a one-dimensional sample and a scalar label to update an index incrementally:

index = cvi.CH()
values = np.empty(len(labels))

for i, (sample, label) in enumerate(zip(samples, labels)):
    values[i] = index.get_cvi(sample, int(label))

The same object may also receive incremental samples after its initial batch call. Feature dimensionality must remain constant throughout the object’s lifetime.

State and Input Rules

All CVI implementations are stateful accumulators. Keep the following rules in mind:

  • Batch data have shape (n_samples, n_features) and incremental samples have shape (n_features,).

  • Labels are arbitrary integer identifiers.

    They need not be consecutive or start at zero.

  • Batch initialization requires at least two distinct labels, and a second batch call on the same object is rejected.

  • Criterion values are 0.0 while an index is not defined, such as before enough clusters have been observed.

    Do not interpret that sentinel as an optimal clustering result.

  • Use a fresh instance when comparing independent datasets or partitions.

See Choosing an Index for differences between indices. In particular, CONN has additional preprocessing and backend requirements described in Using CONN.

Updating a partition

After batch or incremental initialization, a sample can be added with get_cvi and (except for CONN) an existing sample can be removed or two clusters can be merged:

value = index.get_cvi(new_sample, new_label)
value = index.remove(existing_sample, existing_label)
value = index.merge(target_label=20, source_label=10)

These operations update the object in place and return its new criterion value. merge retains the target label and deletes the source label. Removing a cluster’s final sample deletes that label; removing the final sample in the whole index returns the object to its initial empty state.

The package stores sufficient statistics rather than the original dataset. Consequently, the caller must ensure that a sample passed to remove really belongs to the supplied label. Invalid labels, inconsistent samples, changed feature dimensions, and attempts to merge a label with itself raise an error.

Index metadata

Every implementation exposes an info class attribute describing its name, range, and optimization direction:

>>> import cvi
>>> cvi.CH.info
CVIInfo(name='Calinski-Harabasz', name_short='CH', index_min=0.0, index_max=inf, optimality='max')

Use optimality rather than assuming that a larger value is always better.

Acknowledgements

Derivation

The incremental and batch CVI implementations in this package are largely derived from the following Julia language implementations by the same authors of this package:

Authors

The principal authors of the cvi pacakge are: