k1lib.kdata module

Everything related to data transformation and loading. This is exposed automatically with:

from k1lib.imports import *
kdata.FunctionData # exposed

class k1lib.kdata.FunctionData

Bases: object

static main(f: Callable, bs: int = 32, epochs: int = 300)

Constructs 2 dataloaders, train and valid, for a particular function. Example:

trainDl, validDl = kdata.FunctionData.main(torch.exp, 32, 300)
for epoch in range(3):
    for xb, yb in trainDl:
        model(xb)

static exp(bs, epochs)

static log(bs, epochs)

static inverse(bs, epochs)

static linear(bs, epochs)

static sin(bs, epochs)

k1lib.kdata.tfImg(size: Optional[int] = None, /, flip=True) → k1lib.cli.init.BaseCli

Get typical image transforms. Example:

"path/img.png" | toPIL() | kdata.imgTf(224)

k1lib.kdata.tfFloat(t: Union[Iterator[float], torch.Tensor], force=True) → k1lib.cli.init.BaseCli

Suggested float input transformation function. Example:

# before training
data = torch.randn(10, 20) * 100 + 20 # weird data with weird hist distribution
f = kdata.tfFloat(data)

# while training
newData = torch.randn(10, 20) * 105 + 15
newData | f # nicely formatted Tensor, going uniformly from -1 to 1

Parameters

force – if True, forces weird values to 0.0, else filters out all weird rows.

k1lib.kdata.analyzeFloat(l: Iterator[float])

Preliminary input float stream analysis. Example:

torch.linspace(-2, 2, 50) | kdata.analyzeFloat