Tech Overview

At the core of the General System platform is a multi-attribute index and innovative storage features architected for large scale multi-dimensional analysis
‍

Book a Demo

Demo Video

2 minute video

Test Drive

A world of spatiotemporal data

Access data via the dfipy Python SDK or REST API in any language. Batch and streaming spatiotemporal datasets (ID, latitude, longitude, altitude, date/time + 8 optional filter fields) are loaded into a single table which negates the need for data partitioning and multiple table lookups. Queries to identify or count records within a specified area are executed in seconds instead of hours, enabling fast analysis of locations, routes, dwells and co-location.

The GS platform excels in real-time, contextual querying across datasets. Imagine projects identifying anomalies or comparing trends that previously took hours or days, running programmatically every minute!

How does it work?

Batch and live stream data is indexed on ingest at millions of rows per second for immediate querying in real time. The high-density storage architecture enables scaling to hundreds of billions of records on a single instance.

The multi-attribute index on ID, latitude, longitude and time ordering processes queries exceptionally fast, even at massive scale. No need to partition data, no index bloat, no need to recluster and tune.

Continuous, adaptive, background resharding distributes the data workload evenly to mitigate hotspotting and overloads which saves laborious partitioning and tuning.

Thread-per-core architecture with I/O scheduler uses its visibility and control over storage access and cache to reorder, optimise and reduce storage operations to save hardware and maintenance.

More about the platform

Based on a highly optimized database kernel containing multiple technologies and unique architectural features, the platform provides an unparalleled foundation and scalable solution for processing high velocity, high volume spatiotemporal workloads.

State-of-art, thread-per-core architecture, vectorised storage model and user space scheduling of I/O and execution combine with the ability to simultaneously ingest, index and query streaming workloads.

Read The Whitepaper

Based on proven, reliable AWS architecture

Extremely high levels of security with robust user and data access permissions

Visualization via standard industry tools

The platform complements your existing data infrastructure....

Batch data

Streaming data

Real time engine

Index

Query

Event detection

Data processing

Data lakes/ Warehouse

Real time apps

Real time events

Processing Data in General System vs PostGIS

Technology that is purposed-designed for managing spatiotemporal data can provide significant benefits.

PostGIS - based on ingestion of 17 billion AdTech data records on a db.r6g.16xlarge server with 5 metadata fields.
General System based on ingestion of 101 billion AdTech data records on
an i3en.12xlarge server with 6 metadata fields. Costs include ETL and ingestion costs.

Deep dive into the platform's unique architecture

4-dimensional index design enables diverse and unrelated spatiotemporal datasets to be easily aggregated. Run complex spatiotemporal queries such as polygon relationships and geofencing on 10-100 billion records in seconds.

Thread-per-core architecture with I/O scheduler designed for high dimension indexes uses its visibility and control over storage access and cache to reorder, optimise and reduce storage operations.

Continuous, adaptive, background re-sharding distributes the data workload evenly to mitigate overloads and bottlenecks.

A high-density storage architecture creates a non-POSIX file system that coexists with standard Linux environments to bypass traditional scalability and performance limitations. The platform processes petabytes of data with consistent performance across millions of logical files while implementing additional features that conventional file systems do not support.

Check out the speed and scale of the solution with a synthetic dataset of 1.6 million vehicles in London. The data is based on vehicle number plates with historic movements loaded in time order to realistically simulate a real-time feed of 92.4 billion datapoints.

Book a Demo

Demo Video

2 minute video

Test Drive