Payment DataLake Reservoir System

  • Problem Statement
    • Build a single, complete, high quality store of all banking data. 
  • Business Goals
    • For historic reasons the data was spread over a large number of payment and DDA systems. Every system “spoke a different language” in the way it represented data.  In order to perform any kind of reporting, data science, processing (e.g. billing), operations, etc there was a need to consolidate the data.   
  • Implementation 
    • Publishing Framework
      • Implementation of event, message and file publishing into the data lake using Apache Kafka
    • Intelligence Engine
      • Framework (ETL) for refining, transforming and segregating (for restricted countries) using Spark 
      • Spark Scala, Spark SQL, Spark Streaming, Spark DataFrames and Datasets
  •  Service Layer
      • Data consumption via Kafka Topic

Enterprise Application Microservices to integrate various business services like (HR: Workday, Excelity, Benefits)

  • Technologies included Java 8, JAX-RS, Spring Boot, Spring Core, ZUUL, Consul, Redis, OpenID, Connect, JWT, OAuth 2.0, SQL Server, ELK
  • Amazon VPC, Docker

 

HIPAA compliant Connected Medical Devices with AWS IOT

  • Telemetry data gets sent to AWS IoT Core using MQTT 
  • Device metadata gets stored in DynamoDB and device data is stored in S3.
  • Batch ETL done using Amazon Glue and processed data is stored in S3