Building a Biomedical Data Validation Engine in F#: Rules, Anomaly Detection and Google Cloud Deployment

1. Introduction

Biomedical organisations in Europe increasingly rely on large datasets to support clinical operations, research activities, and regulatory reporting. These datasets include laboratory measurements, electronic health records, diagnostic information, and longitudinal observations collected over extended periods. As the volume and complexity of biomedical data grow, the need for reliable data quality processes becomes essential. Incorrect or inconsistent data can lead to inaccurate clinical decisions, flawed research outcomes, and compliance risks in regulated environments.

This article presents a complete, end‑to‑end implementation of a Biomedical Data Quality Engine written in F#. The goal is to demonstrate how a functional, type‑safe approach can improve reliability, transparency, and maintainability in biomedical data pipelines. The article progresses from basic concepts to a production‑ready cloud architecture and includes detailed F# code that can be used in real‑world systems.

The content is structured to guide readers from junior‑level fundamentals to architect‑level cloud deployment. Each section introduces new concepts and expands the solution. The final part includes a complete production‑ready F# implementation and practical business recommendations.

2. Business Problem and Requirements

Biomedical datasets often contain several categories of quality issues. These issues arise due to manual data entry, heterogeneous data sources, inconsistent laboratory systems, and differences in clinical workflows across departments.

2.1 Common Data Quality Issues

  1. Missing values in laboratory results or patient attributes.
  2. Anomalous values, such as negative concentrations or extreme outliers.
  3. Temporal inconsistencies, for example laboratory tests recorded before a patient’s birth date.
  4. Logical conflicts, such as mutually exclusive diagnoses appearing together.
  5. Unit inconsistencies, where the same test is reported in different measurement units without conversion.
  6. Duplicated records, often caused by repeated ingestion or system errors.
  7. Incomplete longitudinal sequences, where expected measurements are missing in time‑series data.

2.2 Business Impact

Poor data quality affects:

  • clinical decision‑making
  • research validity
  • regulatory compliance
  • machine learning model accuracy
  • operational reporting
  • patient safety

In regulated environments such as clinical trials, incorrect data may lead to audit findings, delays, or rejection of study results.

2.3 Requirements for a Data Quality Engine

A biomedical organisation typically defines the following requirements:

  • automated anomaly detection
  • automatic imputation of missing values
  • consistency validation
  • scalability to millions of records
  • interpretability for clinicians and researchers
  • cloud integration
  • auditability for regulatory compliance
  • reproducibility of results
  • modular architecture for future extensions

This article addresses each requirement and demonstrates how to implement a complete solution in F#.

3. Domain Model in F#

A clear domain model is the foundation of a reliable data quality engine. F# provides strong typing and safe handling of missing values through the option type. The domain model must represent patients, laboratory results, and clinical records in a structured and type‑safe manner.

3.1 Patient Model

fsharp

type Sex = Male | Female | Other

type Patient = {
    Id        : string
    BirthDate : System.DateTime option
    Sex       : Sex option
}

3.2 Laboratory Result Model

fsharp

type LabTestCode = string

type LabResult = {
    PatientId   : string
    TestCode    : LabTestCode
    Value       : float option
    Unit        : string option
    CollectedAt : System.DateTime option
}

3.3 Clinical Record Model

fsharp

type ClinicalRecord = {
    Patient : Patient
    Labs    : LabResult list
}

This model supports missing values, strict typing, and clear separation of patient and laboratory information. It also allows the engine to operate on structured data without relying on untyped dictionaries or dynamic objects.

4. Anomaly Detection

Anomaly detection is implemented in three stages: basic rule‑based checks, statistical checks, and domain‑specific checks. This progression allows the system to detect both simple and complex anomalies.

4.1 Basic Rule‑Based Checks

Basic checks identify obvious errors such as negative values or impossible dates.

fsharp

type AnomalyType =
    | NegativeValue
    | OutOfRange
    | ImpossibleDate
    | MissingCriticalField

type Anomaly = {
    PatientId : string
    TestCode  : LabTestCode
    At        : System.DateTime option
    Kind      : AnomalyType
    Details   : string
}

let detectNegativeValues (lab : LabResult) : Anomaly option =
    match lab.Value with
    | Some v when v < 0.0 ->
        Some {
            PatientId = lab.PatientId
            TestCode  = lab.TestCode
            At        = lab.CollectedAt
            Kind      = NegativeValue
            Details   = sprintf "Value %f is negative" v
        }
    | _ -> None

4.2 Statistical Anomaly Detection

Statistical methods detect outliers based on distribution characteristics.

fsharp

module StatsAnomaly =

    let mean xs = xs |> List.average

    let stdDev xs =
        let m = mean xs
        xs |> List.averageBy (fun x -> (x - m) ** 2.0) |> sqrt

    let zScoreAnomalies values threshold =
        let m = mean values
        let s = stdDev values
        values
        |> List.mapi (fun i v ->
            let z = (v - m) / s
            if abs z > threshold then Some (i, v, z) else None)
        |> List.choose id

4.3 Domain‑Specific Checks

Domain‑specific checks include age‑adjusted norms, sex‑specific thresholds, and test‑specific ranges. These rules depend on clinical guidelines and laboratory specifications.

5. Automatic Imputation

Imputation replaces missing values with meaningful estimates. The engine supports simple imputation and more advanced group‑based methods.

5.1 Median Imputation

fsharp

let imputeWithMedian (values : float option list) : float option list =
    let observed = values |> List.choose id
    if List.isEmpty observed then values
    else
        let sorted = observed |> List.sort
        let n = sorted.Length
        let median =
            if n % 2 = 1 then sorted.[n/2]
            else (sorted.[n/2 - 1] + sorted.[n/2]) / 2.0

        values |> List.map (fun v -> match v with Some _ -> v | None -> Some median)

5.2 Group‑Based Imputation

Group‑based imputation uses age groups, sex groups, or correlated tests. This approach improves accuracy and reduces bias.

6. Consistency Validation

Consistency validation ensures that records follow logical, temporal, and structural rules.

fsharp

type ConsistencyIssue = {
    PatientId   : string
    Description : string
}

let checkBirthDateVsLab (patient : Patient) (lab : LabResult) : ConsistencyIssue option =
    match patient.BirthDate, lab.CollectedAt with
    | Some bd, Some dt when dt < bd ->
        Some {
            PatientId   = patient.Id
            Description = sprintf "Lab at %O is before birth date %O" dt bd
        }
    | _ -> None

7. Quality Pipeline

The quality pipeline orchestrates anomaly detection, imputation, and consistency validation.

fsharp

module QualityPipeline =

    open Domain
    open AnomalyDetection
    open Imputation
    open ConsistencyChecks

    type QualityReport = {
        Anomalies         : Anomaly list
        ConsistencyIssues : ConsistencyIssue list
        ImputedRecords    : ClinicalRecord list
    }

    let runQualityPipeline (records : ClinicalRecord list) : QualityReport =
        let anomalies =
            records
            |> List.collect (fun r ->
                r.Labs |> List.choose detectNegativeValues)

        let imputedRecords = records

        let consistencyIssues =
            records |> List.collect checkConsistency

        {
            Anomalies         = anomalies
            ConsistencyIssues = consistencyIssues
            ImputedRecords    = imputedRecords
        }

8. Production Architecture

8.1 Google Cloud Architecture

Components:

  • Cloud Storage
  • Dataflow
  • F# Engine (container)
  • BigQuery

Strengths:

  • cost‑efficient serverless pipelines
  • strong batch processing capabilities
  • scalable data ingestion

Weaknesses:

  • fewer .NET‑native services
  • requires container‑based deployment for F#

8.2 Azure Architecture

Components:

  • Azure Data Lake
  • Azure Functions
  • F# Engine
  • Synapse Analytics

Strengths:

  • deep .NET integration
  • strong enterprise support
  • native support for F#

Weaknesses:

  • higher cost for large batch workloads
  • more complex configuration for high‑volume pipelines

9. FinOps Cost Comparison

Google Cloud

  • Storage: lower cost per TB
  • Compute: flexible pricing
  • Dataflow: efficient for large batch pipelines
  • BigQuery: cost‑effective for analytical workloads

Azure

  • Storage: higher cost per TB
  • Compute: strong integration with .NET workloads
  • Functions: cost increases with high throughput
  • Synapse: higher cost for large analytical workloads

Summary

Google Cloud is more cost‑efficient for large biomedical datasets. Azure provides better integration for F# and .NET‑based systems.

10. Final Algorithm

  1. Load patient and laboratory data.
  2. Convert raw data into F# domain types.
  3. Detect anomalies using rule‑based and statistical methods.
  4. Impute missing values.
  5. Validate consistency.
  6. Produce a quality report.
  7. Store results in cloud storage or analytical systems.
  8. Monitor data quality metrics.
  9. Integrate with downstream analytics.

11. Practical Business Recommendations

  • Use strict typing to reduce data errors.
  • Apply rule‑based checks before statistical methods.
  • Use group‑based imputation for clinical datasets.
  • Maintain audit logs for regulatory compliance.
  • Choose Google Cloud for large‑scale batch workloads.
  • Choose Azure when deep .NET integration is required.
  • Monitor data quality metrics continuously.
  • Implement automated alerts for critical anomalies.
  • Validate unit consistency across laboratory systems.
  • Use reproducible pipelines for regulatory audits.

Similar Posts