Web Scraping / Web Crawling / Data Mining / Browser Automation Expert

Your scraper broke again. The data stopped flowing. Your dashboard shows yesterday's numbers.

I build the systems that don't do that.

Over the past 5+ years, I've built production data pipelines, backend services, and automation systems that process millions of records reliably. My work goes beyond extracting data—I design the infrastructure around it, including task queues, schedulers, APIs, databases, authentication flows, and monitoring, so the entire pipeline keeps running long after deployment.

One of my largest projects powers a pharmaceutical data provider, ingesting millions of products across 30+ vendors on a recurring schedule. The pipeline validates every run with automated quality gates that detect silent failures before corrupted data reaches downstream systems or warehouses.

## Real production problems I've solved

### Pharmaceutical Catalog Ingestion (NDA)

The existing scraper relied on HTML selectors and frontend pagination. During investigation I discovered the site silently returned corrupted data whenever a cookie and header combination became invalid. No exception was raised, only incorrect product counts.

I reverse engineered the underlying product API and rebuilt the pipeline around backend endpoints instead of the frontend.

Result

* HTML selectors eliminated
* Silent corruption removed
* Faster and significantly more reliable extraction

---

Another target triggered an anti bot challenge on nearly every request, rapidly consuming captcha solver credits.

After analyzing the request flow I discovered that a specific warm up request established a valid session and bypassed the challenge for the remainder of the IP session.

Result

* Solver usage reduced dramatically
* Throughput increased by roughly 3x
* Lower operating cost with higher stability

---

A third target exposed no public JSON endpoints and was protected behind a complex frontend.

By identifying the platform as Magento, I reconstructed the catalog directly through its GraphQL layer, bypassing the UI completely.

Result

* Structured data instead of HTML parsing
* Faster collection
* More resilient pipeline

---

### Insurance & Banking Portals (NDA)

Built a backend platform integrating multiple SAML and OAuth authenticated portals, each with different session lifecycles, validation rules, MFA requirements, and request formats.

The platform minimizes browser automation by reverse engineering APIs whenever possible, orchestrates authentication flows, classifies failures by severity, retries recoverable errors automatically, and prevents partial or corrupted data from propagating downstream.

A sample implementation covering an F5 BIG IP APM → ADFS → Cognito OAuth PKCE authentication chain is available on request.

## Beyond scraping

Many projects require much more than collecting data.

I also build the backend services that keep those systems running in production, including

* REST APIs for downstream consumers
* Celery workers for distributed proce

Attachments (Click to Preview)