Activeloop

Deep Lake stores raw data and vectors together, built on a storage format optimized for deep learning applications. That covers two jobs usually split across two systems, storing and searching data plus vectors while building LLM applications, and managing datasets while training deep learning models.

The format holds embeddings, audio, text, video, images, DICOM files, PDFs and annotations rather than embeddings with light metadata on top. It is serverless, computations run client side, and the data stays in your own cloud. Installation is a single pip install of the deeplake package.

Several capabilities follow from that format.

• Multi-cloud storage: uses one API for S3, Azure, GCP, Activeloop cloud, local or in-memory storage, including S3 compatible systems such as MinIO.
• Lazy indexing: keeps media in native compression and lets you slice and iterate it like NumPy arrays, loading only what is needed.
• Dataloaders: feed PyTorch and TensorFlow directly, dataset shuffling included.
• Versioning and lineage: track how a dataset changed, the way version control tracks code.
• Visualization: renders datasets with bounding boxes, masks and annotations in the Deep Lake app.
• Integrations: connect LangChain, LlamaIndex, Weights & Biases, MMDetection and MMSegmentation.

More than one hundred community uploaded image, video and audio datasets, among them MNIST, COCO, ImageNet and CIFAR, can be pulled in within seconds. Deep Lake is used by Intel, Bayer Radiology, Matterport, Red Cross, Yale and Oxford, and universities can claim a free monthly storage and query allowance.