DataFlow is an enterprise-grade data integration platform built in Rust. Design complex pipelines visually, execute them with sub-millisecond precision, and monitor everything in real time.
DataFlow is built around a high-performance Rust engine, a TypeScript visual designer, and a comprehensive monitoring system — all wired together through a unified API gateway.
DAG-based pipeline executor with wave-parallel processing, conditional branching, sub-flows, data lineage tracking, and cron scheduling. Built on Tokio async runtime for maximum throughput.
Drag-and-drop React canvas powered by React Flow. Auto-generated config forms from module JSON Schemas. Version history, diff viewer, one-click rollback, and credential management.
Real-time execution waterfall, structured log viewer, system health indicators, and cancel actions. OpenTelemetry traces, Prometheus metrics, and pre-built Grafana dashboards.
Full REST API with RBAC enforcement, OIDC/SAML/local auth, JWT middleware, and an MCP server for AI assistant integration. All endpoints are typed, audited, and tested.
OIDC, SAML 2.0, and local auth with 16-permission RBAC. Ed25519-signed offline licenses — no call-home required, works in air-gapped networks.
Exposes platform operations as Model Context Protocol tools. AI assistants can list flows, trigger executions, check status, and cancel runs — without custom integration work.
Every module implements the same FlowModule trait — chainable, configurable, and testable. The ecosystem spans file transfer, cloud storage, data processing, B2B/EDI, ERP/CRM, e-commerce, security, and more. Extend further with the Rust or Python SDK.
| Module | Type ID | Description | Status |
|---|---|---|---|
| SFTP Transfer | sftp-transfer | Upload/download via SFTP with key/password auth, glob patterns, post-download actions | Live |
| AS2 Transfer | as2-transfer | Send/receive files via AS2 (EDI over HTTP) with MDN acknowledgment support | Live |
| FTP Transfer | ftp-transfer | Upload/download via FTP/FTPS (explicit and implicit TLS), active/passive modes, glob patterns | In Progress |
| AS4 Transfer | as4-transfer | ebMS 3.0 / AS4 for EU B2B (Peppol, e-SENS), WS-Security signing/encryption, receipt handling | In Progress |
| OFTP Transfer | oftp-transfer | OFTP2 (RFC 5024) for automotive/manufacturing supply chains, TLS, EERP/NERP receipts, restart/recovery | In Progress |
| Module | Type ID | Description | Status |
|---|---|---|---|
| AWS S3 | cloud-s3 | Upload/download/list/delete S3 objects, multipart uploads, server-side encryption, presigned URLs, SQS/SNS triggers | In Progress |
| Google Cloud Storage | cloud-gcs | Upload/download/list/delete GCS objects, IAM or HMAC key auth, lifecycle policies, Pub/Sub event trigger | In Progress |
| Azure Blob Storage | cloud-azure-blob | Upload/download/list/delete blobs, Shared Key or SAS token auth, hot/cool/archive tiers, Event Grid trigger | In Progress |
| Cloudflare R2 | cloud-r2 | S3-compatible API for Cloudflare R2 with zero egress fees, Workers integration | In Progress |
| Module | Type ID | Description | Status |
|---|---|---|---|
| File Transform | file-transform | CSV, JSON, XML, and fixed-width format conversions | Live |
| File Router | file-router | Route data to different branches based on content rules or metadata | Live |
| File Compression | file-compress | ZIP, GZIP, and TAR compression and decompression | Live |
| CSV File | csv-file | Parse, process rows, and generate CSV with per-row filtering and routing | Live |
| XML Processor | xml-process | Parse, transform (XSLT), validate (XSD), XPath extraction, generate. Streaming for large documents | In Progress |
| JSON Processor | json-process | Parse, transform (JSONPath/JMESPath), validate (JSON Schema), merge, split, flatten/unflatten | In Progress |
| Document Processor | doc-process | Read/write DOCX (template fill, mail merge), XLSX (formulas, named ranges), PDF (text extraction, form fill) | In Progress |
| Module | Type ID | Description | Status |
|---|---|---|---|
| PGP Encrypt/Decrypt | pgp-crypto | Encrypt, decrypt, sign, verify with rPGP 0.14. ASCII armor and binary output | Live |
| AES Encrypt/Decrypt | aes-crypto | AES-128/192/256 in CBC, CTR, GCM, CCM modes. PKCS7 padding, configurable IV, Base64 or binary output | In Progress |
| 3DES Encrypt/Decrypt | 3des-crypto | Triple DES in CBC and ECB modes for legacy system interoperability (banking, POS, FIPS 46-3) | In Progress |
| Hash & Checksum | hash-digest | MD5, SHA-1/256/384/512, SHA3, CRC32, Blake2b, Blake3. hash_file, verify_hash, hash_manifest operations | In Progress |
| HMAC | hmac-auth | HMAC-SHA256/384/512/SHA3 for webhook signature verification (Shopify, Stripe, QuickBooks) and B2B integrity | In Progress |
| S/MIME Encrypt/Sign | smime-crypto | S/MIME (RFC 8551) encryption and signing with X.509 certs. Integrates with platform certificate manager | In Progress |
| Module | Type ID | Description | Status |
|---|---|---|---|
| Web API Call | http-request | HTTP requests with Bearer, Basic, ApiKey, OAuth2, and mTLS auth types; template substitution | Live |
| Email Notification | email-notify | Send SMTP email notifications on flow events or pipeline results | Live |
| Database Query | db-query | Execute queries against PostgreSQL, MySQL, MSSQL, Oracle with per-row processing | Live |
| Module | Type ID | Description | Status |
|---|---|---|---|
| EDI Translator | edi-translate | Parse/generate X12 EDI (850, 855, 856, 810, 860, 997, 852, 940, 945) with retailer-specific profiles | In Progress |
| Module | Type ID | Description | Status |
|---|---|---|---|
| QuickBooks Connector | quickbooks-connect | Read/write QuickBooks Online (OAuth2 REST) and Desktop (QBXML) — invoices, POs, items, customers | Live |
| SAP Connector | sap-connect | SAP ERP and S/4HANA via RFC/BAPI or OData APIs. IDocs inbound/outbound, sales orders, material master | In Progress |
| Sage Connector | sage-connect | Sage 50/200/Intacct REST API — invoices, purchase orders, GL entries, inventory, customer/vendor records | In Progress |
| Salesforce Connector | salesforce-connect | Salesforce REST/Bulk API with OAuth2 — SOQL queries, sObject CRUD, bulk import/export, Platform Events | In Progress |
| Module | Type ID | Description | Status |
|---|---|---|---|
| Shopify Connector | shopify-connect | Shopify Admin REST/GraphQL — orders, products, inventory, fulfillments. Webhook-triggered flows | In Progress |
| Instacart Connector | instacart-connect | Instacart Connect API — catalog sync, order ingestion, availability updates, delivery status tracking | In Progress |
| DoorDash Connector | doordash-connect | DoorDash Drive API — delivery creation, status tracking, order integration, menu/catalog sync | In Progress |
| Module | Type ID | Description | Status |
|---|---|---|---|
| Spreadsheet Connector | spreadsheet-connect | Read/write Excel (.xlsx/.xls) and Google Sheets with column mapping and header auto-detection | Live |
| Module | Type ID | Description | Status |
|---|---|---|---|
| Script Runner | script-runner | Execute sandboxed Python or shell scripts within a flow | Future |
| File Watcher | file-watcher | Trigger flows based on file system events | Future |
DataFlow is purpose-built for retail EDI — enabling small and mid-size suppliers to connect their existing tools (QuickBooks, Excel) directly to major retailer EDI requirements over AS2.
The DataFlow engine is built for production. Every execution detail — waves, conditions, sub-flows, lineage — is tracked, stored, and visible.
Independent steps are grouped into waves via DAG topological analysis. Each wave executes concurrently up to a configurable max_parallel limit using Tokio JoinSet.
10 condition variants: Expression, Equals, NotEquals, GreaterThan, LessThan, Exists, All, Any, Not, Always. Template variable resolution from previous step outputs.
Flows can invoke other flows as sub-flows with full parent chain tracking, depth limits, and cycle detection. Enables modular pipeline design at scale.
Every DataProduced, DataConsumed, DataTransformed, SubFlowLink, ExternalSource, and ExternalSink event is tracked. Upstream and downstream traces via recursive CTE queries.
Configurable exponential backoff per step. Fail-fast propagation stops downstream steps on error. Cancellation checks between waves for clean shutdown.
Cron-based flow triggering with timezone support. Preview next run times via API. Full schedule lifecycle management through the UI or API.
DataFlow deploys wherever you need it — cloud-native Kubernetes, major cloud providers via Terraform, or bare-metal Linux and Windows servers.
Production Helm chart with HPA, PDB, NetworkPolicy, ServiceMonitor, Ingress with TLS. Deploy to any K8s cluster.
Terraform modules for EKS + RDS. One command to provision a full production environment.
Terraform modules for GKE + Cloud SQL. Same infrastructure-as-code approach across all clouds.
Terraform modules for AKS + PostgreSQL. Deploy to Azure with the same Terraform patterns.
Interactive and unattended systemd installer. ~50MB binary, minimal dependencies.
NSSM-based Windows service installer. Runs alongside existing Windows infrastructure.