Top 5 Pyspark Jobs in Buffalo Ny

Senior Data Engineer Consultant (AI-Forward, Kafka + Databricks)
Proactive Logic Consulting Inc —United States
- $75 - $105 an hour
- Contract
- Remote
- DevOps
- HIPAA
- SQL
Quick Apply

Check out these similar jobs

Based on your search for pyspark jobs.

Senior Data Engineer Consultant (AI-Forward, Kafka + Databricks)
Proactive Logic Consulting Inc —United States
Daily practitioner with AI coding tools and agents: *Claude Code*, *OpenAI Codex CLI* (and similar). Prompt and instruction design; translate business goals…
$75 - $105 an hour
Quick Apply
Senior Healthcare Integration Consultant - EDI 834 & Eligibility
Proactive Logic Consulting Inc —United States
*Business Process Automation:* Partner with ~15 SMEs to map current-state workflows, document decision trees, and automate repetitive manual eligibility edits.
$75 - $110 an hour
Quick Apply

Explore jobs in cities nearby

Check out jobs in cities near your current location, Columbus

Data Engineer
Proteam Solutions —Dublin, OH
We are seeking an experienced Senior Data Engineer with deep expertise in Databricks to design, develop, and optimize enterprise-scale data pipelines.
Quick Apply
Lead ETL Developer- Enterprise Data Warehouse (EDW)
Huntington Bank —Columbus, OH
As a technical leader, you will partner across engineering, architecture, QA, and business teams to deliver high-quality, reliable data solutions.
$70,000 - $140,000 a year

Show more jobs in cities nearby

I want to receive the latest job alert for pyspark in buffalo ny

By signing in to your account, you agree to SimplyHired's Terms of Service and consent to our Cookie and Privacy Policy.

Explore jobs in more locations

pyspark jobs near buffalo ny

Design Consultant jobs in Chicago, IL

Senior Engineering Consultant jobs in Tampa, FL

Senior Marketing Manager jobs in Remote

Belmont Village Senior Living jobs in La Jolla, CA

Senior Data Engineer Consultant (AI-Forward, Kafka + Databricks)

Proactive Logic Consulting Inc
United States

Quick Apply

Job Details

Contract
$75 - $105 an hour

Qualifications

AI models
Cost management
Data model design
Data quality checks
Commercial use (data warehousing systems)
Performance tuning
Databricks
Data Integration (Data management)
Cloud Logging
Requirements design
Achieving HIPAA compliance
Tooling
Infrastructure as Code (IaC)
Triage
DevOps
Data quality assurance
Spark
System performance optimization
HIPAA
AI platforms (beyond public GPTs)
Machine learning projects
Schema design
Business consulting
Snowflake
Prompt engineering
SQL
Development environments
Azure Synapse Analytics
Reducing cloud infrastructure costs
Data Architecture Design (Architecture design skills)

Full Job Description

Job post summary

Date posted: September 24, 2025

Pay: $75.00 - $105.00 per hour

Job description:

Who We Are

Proactive Logic Consulting INC is a boutique technology consulting firm specializing in:

Innovation, zero to one projects
Assessments & Roadmaps
Process Automation and AI
Azure Cloud Migrations
App Modernization (including Low-Code solutions)

We bring together top-tier independent consultants—experts in their craft who thrive on flexibility and autonomy. Our consultants have a strong customer-obsessed mindset, high EQ, and deep technical skills. We’re entrepreneurial at heart and fully support individuals who are simultaneously building their own product ventures.

What We’re Looking For

We need an experienced Senior Data Engineer Consultant (AI-Forward, Kafka + Databricks) who can design scalable streaming and batch data platforms—and integrate Large Language Models (LLMs) and agentic workflows into real analytics and data operations. This role is a corp-to-corp, 1099 capacity. If you deliver high-quality outcomes, work independently, and leverage AI tools to multiply impact, we’d love to talk.

Key Requirements

10+ Years in Data Engineering
Building and operating reliable data platforms, ELT/ETL pipelines, and lakehouse architectures.
Strong data modeling (medallion/bronze-silver-gold, dimensional/star schema, semantic layers).
Kafka Expertise
High-throughput, fault-tolerant streaming design (partitions, consumer groups, idempotency, EOS semantics).
Kafka Streams/Flink/Spark Structured Streaming; Connectors & Schema Registry; CDC (e.g., Debezium).
Confluent Cloud or self-managed; security (SASL/SCRAM, mTLS), governance, and capacity planning.
Databricks / Spark
PySpark/Scala/Spark SQL; Delta Lake, Delta Live Tables, Structured Streaming, MLflow, Unity Catalog.
Job orchestration (Workflows), cluster sizing/tuning (incl. Photon), CI/CD, cost optimization.
Azure Data Platform
Azure Databricks, ADLS Gen2, Event Hubs (Kafka), Data Factory/Synapse pipelines, API Management, Key Vault.
Azure DevOps/GitHub, IaC (Terraform/Bicep), Observability (App Insights/Log Analytics/OpenTelemetry).
AI-Forward Data Engineering
Daily practitioner with AI coding tools and agents: Claude Code, OpenAI Codex CLI (and similar).
Prompt and instruction design; translate business goals into structured prompts, tools, and context.
LLM integration for data: NL-to-SQL, documentation/enrichment, catalog search, semantic layers, and RAG.
Embeddings and vector search (Azure AI Search, pgvector, Pinecone or equivalent) over lakehouse/catalog.
Agentic workflows for ops: incident runbooks, pipeline triage, schema-change diffing, and code reviews with human-in-the-loop.
Safety/compliance: PII/PHI handling, redaction, guardrails/moderation, data residency, secrets (Key Vault), HIPAA alignment.
Measurement & cost: evals, latency/throughput tradeoffs, token/cost tracking, model/version pinning and resilient fallbacks.
Governance & Quality
Unity Catalog, lineage (OpenLineage/DataHub), data quality (Great Expectations/Deequ), SLAs/SLOs.
Healthcare/Pharmacy Domain (Preferred)
Experience in healthcare or pharmacy; understanding of HIPAA and PHI constraints.
Consulting & EQ
Excellent client-facing communication and requirements discovery; outcome-focused and pragmatic.
Entrepreneurial Mindset
Think like an owner; innovate and deliver measurable value.
Business Structure
Must have an active LLC or S-Corp; 1099 Only (Corp-to-Corp).

What You’ll Do

Architect & Design: Lakehouse and streaming architectures on Azure with Kafka + Databricks.
Build Streaming & Batch: Kafka ingestion (incl. CDC), DLT pipelines, ELT to Delta, curated gold datasets.
AI-Integrated Data: Implement NL-to-SQL, semantic catalog search, RAG over documentation/metadata, and automated doc generation.
Operate & Optimize: Establish observability, data quality checks, cost/latency budgets, and performance tuning.
Govern & Secure: Apply Unity Catalog policies, lineage, masking/row-level security, and HIPAA-aligned practices.
Client Engagement: Translate business needs into data roadmaps; present trade-offs in cost, performance, and accuracy.
Continuous Improvement: Track model/platform updates; evolve data and AI architecture as capabilities improve.

Preferred Tech & Tooling

Kafka, Confluent Cloud, Kafka Connect, Schema Registry, Debezium, Flink (plus)
Databricks: Spark, Delta Lake, Delta Live Tables, Workflows, Unity Catalog, MLflow, Repos/dbx, SQL
Orchestration: ADF/Synapse Pipelines, Databricks Workflows, Airflow
Data Quality & Lineage: Great Expectations, Deequ, OpenLineage, DataHub, Azure Purview
Azure: ADLS Gen2, Event Hubs (Kafka), Functions, API Management, Key Vault, App Insights/Log Analytics
LLM platforms: Azure OpenAI, OpenAI, Anthropic; vector: Azure AI Search, pgvector, Pinecone
Orchestration libs: Semantic Kernel, LangChain/LangChain4j, LlamaIndex
Agentic tooling & IDEs: Claude Code, OpenAI Codex CLI, GitHub Copilot/Cursor (or similar)

Why Join Proactive Logic Consulting?

Flexible Engagements: Autonomy in how you work and deliver.
Top-Tier Peers: Senior experts in cloud, data, and AI.
Impactful Projects: Healthcare and modernization with measurable business outcomes.
AI-Forward Culture: Operate at the leading edge of agentic data engineering.
Entrepreneurial Community: Connect with builders and founders.

How to Apply

If this describes your ideal consulting engagement and you meet the qualifications, please send:

Your resume or consultant profile highlighting data engineering, Kafka, Databricks, and AI/LLM experience.
A brief description of your LLC or S-Corp (name, location).
Links to data/AI-forward work (streaming pipelines, notebooks, DBX repos, MLflow runs, or technical write-ups).
A short statement on how you use AI tools (Claude Code, OpenAI Codex CLI, etc.) to accelerate delivery while maintaining quality and safety.

Job Type: Contract

Compensation Package:

1099 contract

Experience:

Data engineering: 10+ years (Required)
Kafka (Confluent or equivalent): 4+ years (Required)
Databricks/Spark: 3+ years (Required)
AI/LLM integration in data workflows: 1–2+ years (Preferred)
Consulting: 5+ years (Required)

Work Location: Remote

Job Type: Contract

Pay: $75.00 - $105.00 per hour

Expected hours: 20 – 40 per week

Work Location: Remote

Quick Apply

Refine Your Search

pyspark jobs in buffalo ny

Senior Data Engineer Consultant (AI-Forward, Kafka + Databricks)

Check out these similar jobs

Explore jobs in cities nearby

I want to receive the latest job alert for pyspark in buffalo ny

Explore jobs in more locations

Job Seeker Tools

Employer Tools

Browse

Stay Connected