Why We Build Heavy Data Pipelines in F#, Not Python

Introduction: The Monopoly of the Snake and the Reality of Production

If you open any modern tutorial, cloud documentation, or generic technical blog about Data Engineering, you will see exactly one programming language. Python is presented as the absolute standard, the golden hammer, and the ultimate answer to every data problem.

This monopoly makes sense on the surface. Python has a massive ecosystem, a low barrier to entry, and libraries like Pandas or PySpark that can do almost anything. For Data Science, Machine Learning, and ad-hoc data exploration, Python is genuinely unmatched.

However, Data Engineering is not Data Science.

Data Science is about discovering insights in a static Jupyter Notebook. Data Engineering is about building reliable, highly concurrent, fault-tolerant backend systems that process terabytes of unpredictable, malformed, and aggressive web traffic 24 hours a day, 7 days a week. When you transition from exploring data to streaming thousands of raw JSON logs per second into Google BigQuery, the rules of the game change completely.

In heavy production environments, Python’s dynamic nature, high memory overhead, and lack of true multithreading stop being minor inconveniences. They become critical points of failure and massive sources of infrastructure costs.

This research article explains the architectural shift. It proves with facts, memory mechanics, and performance benchmarks why we choose F#—a functional, strictly typed language on the .NET platform—to build mission-critical data pre-processing pipelines. We will look at memory utilization, domain modeling, concurrency, and the ultimate financial impact on cloud billing.

Chapter 1. Mythbusting: The “Python is Fast Enough” Illusion

Before we dive into F#, we must destroy the most common myth in the modern data industry: “Python is fast because libraries like Pandas and NumPy are written in C.”

This statement is technically true but architecturally misleading. Yes, if you have a perfectly clean, flat CSV file loaded into memory, Pandas will perform vector operations in C at incredible speeds. But real-world data engineering almost never looks like this.

In the real world of web analytics and log processing, data arrives as a continuous stream of deeply nested, messy, and unpredictable JSON payloads.

The Python Penalty

When you process a stream of API requests or Kafka messages in Python, you are not using vectorized C operations. You are creating Python dictionaries. You are parsing JSON into native Python objects. This is where the disaster begins.

Myth 1: Python variables are lightweight.

Reality: In Python, everything is a full object. A simple integer like 42 is not just 4 bytes of memory. It is a full C structure (PyObject) containing a reference count, a type pointer, and the value size. In a standard 64-bit environment, a single integer in Python consumes 28 bytes. A simple string has an overhead of roughly 50 bytes.

When you parse a payload with 100 fields, and you process 10,000 payloads per second, this object overhead creates massive memory pressure. Your cloud container needs gigabytes of RAM just to hold basic numbers and text.

Myth 2: Type Hints solve Python’s typing problems.

Reality: Python introduced Type Hints (def process(data: dict) -> list:), which helps your IDE give you better autocomplete. But these hints do absolutely nothing at runtime. The Python interpreter ignores them. If a frontend developer accidentally sends a string "null" instead of a boolean False, Python will happily accept it, pass it through your pipeline, and crash at the very end when trying to insert it into BigQuery.

Chapter 2. The F# Advantage: Functional-First and Architecturally Sound

F# is a functional-first programming language that runs on the highly optimized .NET Core runtime (now simply .NET). It combines the mathematical strictness of functional programming with the raw performance of a compiled, statically typed enterprise framework.

Why does this matter for data pipelines?

1. Immutability by Default

In data engineering, state mutation is the enemy of reliability. When multiple threads are processing data simultaneously, changing the value of a variable causes race conditions and corrupted data.

In Python, every dictionary is mutable. Any function can accidentally change a value inside your data stream.

In F#, variables (called bindings) are immutable by default. Once a log is parsed into a record, it cannot be changed. To modify the data, you must return a new, transformed copy. This eliminates an entire category of runtime bugs.

2. The Type System as a Unit Test

F# does not just have types; it has an expressive algebraic type system. If your code compiles in F#, there is a very high statistical probability that it will execute without runtime type errors. The compiler forces you to handle every possible edge case.

Let us look at a practical example: Handling missing data.

In Python, missing data is usually represented by None. This leads to the infamous AttributeError: 'NoneType' object has no attribute 'x' that crashes pipelines at 3:00 AM.

In F#, the concept of null is strongly discouraged. Instead, F# uses the Option type. A value is either Some(value) or None. The compiler physically prevents you from accessing the value without explicitly writing logic for the None scenario.

F#

// F# Option Type Example
type WebEvent = {
    SessionId: string
    UserId: string option // This field might not exist
}

let processEvent (event: WebEvent) =
    match event.UserId with
    | Some id -> printfn "Processing known user: %s" id
    | None -> printfn "Processing anonymous session."

If you forget to write the None case, the F# compiler will throw an error and refuse to build the application. The bug is caught on the developer’s laptop, not in the production cloud.

Chapter 3. Domain Modeling: Guaranteeing the BigQuery Schema

One of the most complex tasks in building pipelines is schema validation. BigQuery is a strict database. If you define a column as an INTEGER, and your pipeline sends a FLOAT, the BigQuery Storage Write API will reject the entire batch.

In a Python pipeline, developers usually rely on external libraries like Pydantic or Marshmallow to validate dictionaries. This requires parsing the JSON into a dictionary, then running a validation function (which takes CPU time), and then converting it again.

In F#, the domain model is built into the language itself using Records and Discriminated Unions (DUs).

Discriminated Unions vs. Python Strings

Imagine your web analytics system tracks different types of events: page_view, purchase, and add_to_cart.

The Python Way:

You store the event type as a simple string.

Python

event_type = payload.get("event_type")
if event_type == "purchase":
    process_purchase(payload)
elif event_type == "page_view":
    process_page_view(payload)
else:
    # Hope we don't get an unexpected string
    pass

If a frontend developer makes a typo and sends "purchse", Python will silently fall into the else block and the data is lost forever.

The F# Way:

You define a Discriminated Union. This restricts the data to an exact, mathematically finite set of possibilities.

F#

type EventType =
    | PageView
    | Purchase of float // Purchase must contain a value
    | AddToCart of string // AddToCart must contain a product ID

let routeEvent (eventType: EventType) =
    match eventType with
    | PageView -> 
        // Process logic
    | Purchase amount -> 
        // Process logic, amount is guaranteed to be a float
    | AddToCart productId -> 
        // Process logic, productId is guaranteed

If you add a new event type to the EventType definition but forget to update the routeEvent function, the F# compiler will instantly highlight the exact line of code and warn you about incomplete pattern matching. Mathematical predictability replaces guesswork.

Chapter 4. Memory Management and Throughput: The Benchmark

To prove the superiority of the compiled functional approach, we must look at a realistic heavy-load scenario.

The Test Scenario:

We have an Apache Kafka stream (or a Google Cloud Pub/Sub topic) delivering 20,000 raw JSON web logs per second. The pipeline must:

1. Receive the JSON string.
2. Deserialize it into a strongly typed object.
3. Clean and format timestamps.
4. Filter out bot traffic (invalid sessions).
5. Buffer the data into chunks of 5,000 records.
6. Push the chunks to BigQuery.

The Memory Footprint Comparison

When Python deserializes 20,000 JSON payloads, it creates 20,000 large dictionary objects in the memory heap. The Python Garbage Collector (GC) uses reference counting. When objects are constantly created and destroyed at high speed, the Python GC struggles to keep up, leading to memory spikes. To handle this load without crashing, a Python container often requires 4 GB to 8 GB of RAM.

F# runs on the .NET runtime, which has one of the most advanced Garbage Collectors in the software engineering world. Furthermore, F# allows you to define data models as Structs (Value Types) instead of Classes (Reference Types).

Value types are allocated on the Stack, not the Heap. They are destroyed instantly when the function finishes, completely bypassing the Garbage Collector.

Metric	Python (Pandas / Dicts)	F# (.NET 8)	Why the difference?
Object Overhead	~150-200 bytes per nested dict	0-16 bytes (using structs)	Python stores type data dynamically. F# types are resolved at compile time.
Garbage Collection	Reference Counting (Heavy pauses)	Generational GC / Stack allocation	F# avoids heap allocation for small temporary data processing.
RAM Required (20k req/sec)	~ 4.0 GB	~ 250 MB	Python caches massive dynamic structures. .NET streams data byte-by-byte.
Startup Time	~ 1-3 seconds	~ 50 milliseconds (AOT compiled)	Python interprets code. F# compiles directly to machine code via Ahead-of-Time (AOT).

This is not a marginal improvement. F# uses roughly 16 times less memory to process the exact same workload.

Chapter 5. Concurrency and Parallelism: Breaking the GIL

Data engineering is fundamentally an I/O bound problem. You are constantly waiting for networks: downloading data from an API, reading from a message queue, and uploading to a database.

To process data fast, you need concurrency (doing multiple things at once) and parallelism (using multiple CPU cores simultaneously). This is where Python experiences its most famous architectural failure: the Global Interpreter Lock (GIL).

The Python Concurrency Trap

The Python GIL is a mutex that prevents multiple native threads from executing Python bytecodes at once. This means that even if you deploy a Python script on a massive server with 64 CPU cores, a standard multi-threaded Python application will only ever use 1 core.

To bypass the GIL, Python developers are forced to use the multiprocessing library. Instead of spawning lightweight threads, Python creates entirely separate processes.

The problem? Processes do not share memory. If Process A parses the data and wants to send it to Process B for uploading to BigQuery, Python must serialize the entire dataset using a library called Pickle, send it through an inter-process pipe, and deserialize it on the other side. This serialization overhead destroys any performance benefits you gained by using multiple cores.

The F# Asynchronous Workflow

F# and the .NET runtime do not have a GIL. They offer true, native, OS-level multithreading.

F# treats concurrency as a first-class citizen through its async { } computation expressions and integration with the .NET Task Parallel Library (TPL). When an F# pipeline waits for BigQuery to respond, the thread is instantly released back to the Thread Pool to process other incoming JSON payloads.

F#

// F# True Asynchronous Stream Processing
let processBatchAsync (batch: WebEvent list) =
    async {
        // Map the batch to BigQuery rows
        let rows = batch |> List.map convertToBqRow
        
        // Asynchronously stream to BigQuery without blocking the CPU thread
        let! response = bigQueryClient.InsertRowsAsync(rows) |> Async.AwaitTask
        
        match response.Status with
        | Success -> return ()
        | Error e -> logError e
    }

// Process 10 batches completely in parallel using all available CPU cores
let runPipeline (batches: WebEvent list list) =
    batches
    |> List.map processBatchAsync
    |> Async.Parallel
    |> Async.RunSynchronously

In F#, Async.Parallel distributes the work seamlessly across all available CPU cores. There is no Pickling, no inter-process communication overhead, and no memory duplication. Data flows through the CPU cache efficiently.

Chapter 6. Architectural Integration: The BigQuery Storage Write API

When dealing with BigQuery, inserting data efficiently is an art. Many Python developers use the standard google-cloud-bigquery library and call client.insert_rows_json(). Under the hood, this uses the older REST API. It is slow, prone to timeouts, and charges you for streaming inserts.

The modern standard for heavy pipelines is the BigQuery Storage Write API. It uses gRPC (a high-performance RPC framework) to stream data directly into BigQuery’s internal storage nodes using Protocol Buffers (protobuf) binary serialization.

Implementing gRPC in Python can be painful due to the dynamic typing and the GIL locking up during heavy binary serialization.

Because F# runs on .NET, it has native, ultra-fast support for gRPC and Protocol Buffers. We can define our BigQuery schema as a Protobuf contract. The F# compiler will automatically generate strongly typed classes for this contract.

1. F# receives the JSON web log.
2. The JSON is parsed into a strict F# Record.
3. The Record is mapped directly into a compiled Protobuf binary message.
4. The binary stream is pushed to BigQuery over a multiplexed HTTP/2 gRPC channel.

Because we are bypassing JSON strings entirely during the upload phase, the network payload size is reduced by up to 60%, and BigQuery ingestion speeds increase by orders of magnitude. F# handles the multiplexed connections flawlessly thanks to the .NET connection pool, ensuring we do not exhaust the cloud instance’s TCP ports.

Chapter 7. FinOps: The Financial Reality of the Architecture

Why does all this low-level memory and CPU optimization matter to a business? Why should a CTO care if a developer uses Python or F#?

Because in modern Cloud Computing, architecture dictates the financial bill. Let us translate technical metrics into Cloud FinOps.

Assume we are deploying our data pipeline to Google Cloud Run (a serverless container platform). We are processing a constant stream of 50 million events per day.

Python Deployment (The Heavy Way)

Because Python is memory-heavy and single-threaded (due to the GIL), a single container cannot process many concurrent requests. To prevent Out-Of-Memory (OOM) crashes, we must configure the Python Cloud Run instance with:

• CPU: 2 Cores
• Memory: 4 GB RAM
• Concurrency: Max 10 requests per container.Because the container is inefficient, Cloud Run will automatically scale up to 30 simultaneous instances to handle the traffic spike during peak hours. You pay for all 30 heavy instances.

F# Deployment (The Lean Way)

F# compiles to a highly optimized Linux binary (AOT compilation). It is natively multithreaded and uses almost zero heap memory for this task. We can configure the F# Cloud Run instance with:

• CPU: 1 Core
• Memory: 512 MB RAM
• Concurrency: Max 250 requests per container (it handles this easily).Because a single F# instance can chew through hundreds of concurrent requests simultaneously without crashing, Cloud Run will only need to scale up to 2 or 3 instances to handle the exact same peak traffic.

The Cost Calculation

You are paying for significantly smaller instances, and you need far fewer of them. The F# pipeline will often cost 70% to 85% less to operate per month on Google Cloud Run compared to a standard Python implementation.

Strict typing and compiled functional code are not just academic exercises for programming nerds. They are direct FinOps optimization strategies.

Chapter 8. Mythbusting Revisited: “But Python is Easier to Write!”

The final defense of Python is always about developer velocity. “Python is easier to write. I can deploy a script in an hour. F# requires learning a complex type system.”

This is the classic trap of local optimization versus global optimization.

Yes, writing the first 100 lines of a Python script is faster. But Data Engineering is not about the first hour of writing code. It is about the next 3 years of maintaining it.

When the business logic changes, or when the upstream API modifies a JSON field, a dynamic Python script will fail silently or crash in production. The developer will spend hours debugging logs to find where the NoneType error originated.

In F#, refactoring is fearless. If you change a field in your core domain model, the compiler immediately breaks the build and highlights every single line of code across your entire repository that needs to be updated to support the new logic. The time you “lose” designing types upfront is paid back a thousand times over during maintenance, debugging, and incident response.

Conclusions: The Verdict on Heavy Data Processing

We do not build heavy data pipelines in F# because we hate Python. We use Python extensively where it belongs: in Vertex AI for training machine learning models, in BigQuery ML, and in isolated data science notebooks.

But when the task is to stand between a chaotic stream of millions of raw web events and the strict, expensive architecture of an enterprise data warehouse, Python is the wrong tool for the job.

1. Memory Efficiency is Non-Negotiable: Processing data efficiently requires precise control over memory allocation. Python’s dynamic object model creates massive overhead. F# value types and struct allocations keep the memory footprint tiny.
2. True Concurrency is Mandatory: You cannot build a high-throughput pipeline on a system that locks the CPU to a single thread. F# and .NET provide true, OS-level asynchronous parallelism without the massive serialization penalty of Python’s multiprocessing.
3. Compile-Time Safety Saves Production: Catching errors during compilation is infinitely cheaper and safer than catching them at runtime. F#’s Discriminated Unions and Option types eliminate entire categories of production bugs.
4. FinOps is Architecture: A language that uses less RAM and CPU directly translates to cheaper cloud bills. Serverless container platforms reward efficient code with lower costs.

Mathematical strictness, predictable memory usage, and fearless refactoring are the foundations of reliable Data Engineering. This is why we choose F#.

From the Author’s Desk

Building technical infrastructure forces you to confront your own biases. For years, the industry has pushed a narrative that data engineers must use Python. I have watched talented teams deploy Python pipelines that looked beautiful in the repository but required constant restarts, massive memory limits, and midnight debugging sessions just to keep the data flowing.

When you transition a core parsing service from a dynamic language to a compiled functional language like F#, the first few days feel frustrating. The compiler is aggressive. It refuses to let you cut corners. It forces you to define exactly what happens if a field is missing, if an integer is too large, or if a network call times out.

But once the code compiles, something magical happens. It just runs. It processes terabytes of data, the RAM graph stays perfectly flat, and the cloud billing alerts stop triggering. You stop managing server failures and finally start focusing on actual data architecture. In the end, the ultimate goal of an engineer is not to write popular code; it is to build systems that let you sleep peacefully at night. F# delivers exactly that.