lukasrozado

Data Engineer — Financial Data Ingestion & Reconciliation

I've been investing in crypto since 2015 — that early conviction is what got me into Transfero. Since mid-2025 I've been building the data infrastructure for that institutional crypto/fintech operation: medallion warehouses, multi-chain (EVM/Solana) reconciliation, and BaaS settlement pipelines processing ~300K transactions/month, all in production. Alongside that, I run Augren solo: a market intelligence terminal for esports, in development since 2020. Core stack: Python, PostgreSQL, dbt, Dagster, Docker, Terraform.Every provider payload is persisted verbatim, before any typing or transform.
Typed columns are a convenience layered on top, never a replacement.
That rule exists for one reason: a number is only as trustworthy as the record
of how it got there. If nothing preserves what the source actually sent, you
cannot audit the number, and you cannot trust the code that produced it either.
Everything else lineage, medallion layering, reconciliation, idempotent
reprocessing follows from it.
At Transfero:
- A serverless data lake on Azure Functions ingesting ~300K transactions/month
across institutional sub-accounts, with a header-aware rate limiter that has
produced zero provider bans, and a PostgreSQL COPY + temp-table loader instead
of ORM inserts.
- A multi-chain ledger integrator across 15+ networks (EVM and Solana) using raw
RPC extraction and reverse block-scanning, with continuous auto-reconciliation
against live nodes and a fail-closed gate on any divergence.
- Bronze/Silver/Gold medallion models delivered end-to-end, plus checkpointed
state per atomic unit exact resume from the point of failure, idempotent by
the checkpoint + upsert pair.
- Platform modernization: Dagster for orchestration (same category as Airflow —
DAGs, sensors, backfill, retries), dbt on PostgreSQL, Docker, Terraform.

Outside work I run two systems in production, alone. Augren captures market and telemetry data 24/7 and publishes a warehouse by atomic pointer swap, so a failed publish serves the previous good snapshot stale, never corrupt. Migrating it between infrastructures without stopping capture required a single-writer interlock: new writers refuse to start unless explicitly enabled, so at no instant did two writers touch the same file. Verified by golden-master, SHA256-identical on both sides. Decodis loads ~206M public-registry records with PySpark and COPY at 200–500K rows/s, on a hexagonal architecture with Prometheus and CI.

Core stack: Python (Asyncio, Pandas, FastAPI) · SQL / PostgreSQL · dbt · Dagster ·
DuckDB · PySpark · Parquet · Docker · Terraform · Azure · AWS · Cloudflare Workers.
Portfolio and write-ups: https://lukasrozado.pages.dev/

Experience: 5 years

Yearly salary: $110,000

Hourly rate: $70

Nationality: 🇧🇷 Brazil

Residency: 🇧🇷 Brazil

Experience

Data Engineer

Transfero

2025 - 2026

Audited the inherited data (no raw layer, no lineage, hand-edited values), proved it unreliable, and rebuilt governed Bronze/Silver/Gold medallion warehouses from scratch, orchestrated on a Dagster monorepo with dbt, Docker, and Terraform. Built a rate limiter for a crypto exchange's shared API quota that reads the real consumed-weight header on every response instead of estimating, with state shared across processes. A carousel deprioritizes idle accounts so quota goes to whoever changed. Backfilled full history back to 2019 across tens of millions of records/year at >99.9% uptime, ending the IP-ban pattern that predated it. Replaced a third-party multi-chain indexer with direct RPC extraction across 15+ blockchains (EVM, Solana). A self-healing engine re-extracts any block range whose event count diverges from the chain, and closes the books against the real accounting invariant, on-chain supply from mint minus burn rather than a row count, including full reconciliation of the BRZ stablecoin. Built a custody-integration orchestrator where each regional workspace runs isolated with its own vault-backed credential. A state machine switches on its own between backfill and incremental capture with a 5-minute overlap at the handoff, while a lighter auditor re-scans only transactions still pending from the oldest open date. Fixed a settlement pipeline where financial payloads with native lists broke PostgreSQL's bulk loader: built a sanitization engine converting lists to Postgres arrays before every COPY, and flattened nested payloads fully at the Silver layer so an audit is one SELECT, no join. Designed a hybrid ingestion pipeline for a high-volume, unreliable source: an API high-water mark for near-realtime capture, plus a daily CSV anti-join recovering the pagination-drift gaps the API alone missed. A bounded producer-consumer queue decoupled HTTP calls from writes. Indexes built with CONCURRENTLY never blocked production. Merged two settlement pipelines that had grown apart into one banking-ledger platform, each domain keeping its own isolated cursor and watermark. For endpoints with no native date filter, a watermark early-exit stops reads on its own once history is covered, no full reprocess and no missed record. Balances refresh at two speeds, a 15- minute delta and a daily full pass, while a separate auditor recovers late status changes

Founder & Solo Engineer

Augren

2020 - 2026

Skills

big-data

crypto

data-science

dataops

dba

postgres

python

sql

english

portuguese

Create Profile Hire Data Scientists