Expert insights on data engineering
Full-refresh ETL against a vendor-hosted SCADA source is the easy default and the wrong one. Here's why backfills push pipelines toward full refresh, what it actually costs the source, and the layered incremental pattern (overlap window, updated_at pass, reconciliation, planned deep pulls) that gets the same correctness without the call from the vendor.
Most operators are sitting on years of SCADA history that never made it into their analytical systems. The data is rich, the integration is unglamorous. Here's why the gap persists, what a real SCADA-to-warehouse pipeline looks like, and the volume, downsampling, and reconciliation decisions you have to make on the way.
Field tickets, run tickets, JIBs, and service invoices still arrive at most mid-size operators as paper or PDFs. Here's why the digitization gap has persisted, what a real ingestion pipeline looks like, and where LLM-assisted extraction earns its keep versus where it just creates silent errors in close.
What does a real PPDM ingestion pipeline actually look like? Here's the architecture: Airflow DAGs pulling from OCC and other source systems, landing normalized data into PPDM-aligned PostgreSQL tables, and DuckDB handling the analytical layer on top. Design decisions, common mistakes, and what a working monthly cycle looks like.
Most Oklahoma operators are still pulling Oklahoma Corporation Commission data by hand every month. There's no technical reason for that. Here's what a proper OCC ingestion pipeline looks like, and what it takes to get one running.
We put our entire SDLC in git. Requirements, decisions, task assignments, everything. Then we cancelled standup. Nobody complained. OK, I complained, which is apparently how you get assigned the blog post about it.
A practical reference for data pipeline patterns: loading strategies, slowly changing dimensions, change data capture, Lambda, Kappa, and Medallion architecture, reliability fundamentals like idempotency and atomic swap, and orchestration patterns.
The hyperscale data platform pitch is compelling, but it was designed for a different customer. Here is an honest look at where open source tools like PostgreSQL, DuckDB, Airflow, and dbt outperform proprietary platforms for most Oklahoma organizations, and when the proprietary option is actually the right call.
Stop coupling DAGs by time or ExternalTaskSensor. Airflow's dataset scheduling lets you wire pipelines together through the data they produce and consume, so the right DAGs run at the right time without the fragility.
Oklahoma energy companies are sitting on enormous amounts of data spread across systems that were never designed to talk to each other. PPDM gives you a standard. Data engineering makes it actually work.
We've seen a trend of small-to-mid size Oklahoma businesses outgrowing their data setup. Here's how to tell if it's time to bring in a real data engineer.
If your Airflow Variables, Connections, and secrets only exist in the UI or someone's memory, you don't have a config strategy, you have a time-bomb. Here's how to actually fix that.
Learn how to build portable, testable data pipelines by containerizing your ETL logic and using Airflow purely as a scheduler, keeping your code independent from any specific orchestration tool.
Stop fighting for inbound VPN access. Put your Airflow workers where the data lives and let them call home.
This guide takes an existing Meltano proof-of-concept and elevates it by using a true data source and target database. We will be working with public REST API endpoints as an extractor, and we will use PostgreSQL as a loader.
Containerize a Meltano EL pipeline with Docker to get a reproducible, self-contained workflow that produces a JSONL artifact.