[Hiring] Big Data Platform Engineer @Two95 International Inc.

Role Description

• Manage and support AKS-based Big Data platform infrastructure.
• Administer Apache NiFi, Azure Event Hub, and Azure Functions.
• Implement and maintain CI/CD pipelines using Azure DevOps.
• Monitor platform health using Azure Monitor and Log Analytics.
• Troubleshoot platform, networking, and infrastructure issues.
• Implement infrastructure automation through Terraform and Bicep.
• Ensure platform security, governance, and compliance.
• Collaborate with Data Engineering teams for platform enablement.
• Perform upgrades, patching, capacity planning, and optimization.
• Support production incidents and platform reliability initiatives.

Qualifications

• 8+ years of overall IT experience.
• 5+ years of experience in Cloud Infrastructure Engineering.
• 4+ years of experience working with Kubernetes platforms.
• 3+ years of experience supporting enterprise-scale Data Platforms.
• 3+ years of experience in DevOps/SRE practices.
• 2+ years of experience working with Azure Data Services.

Requirements

• Strong hands-on experience in:
• Apache NiFi
• Azure Event Hub
• Azure Functions
• Enterprise-scale Data Platform Administration
• Azure Kubernetes Service – AKS:
• AKS cluster deployment and administration
• Cluster upgrades and patch management
• Node pool management
• Ingress Controllers, including NGINX and Application Gateway
• RBAC implementation
• AKS security hardening
• Autoscaling using HPA, VPA, and Cluster Autoscaler
• Persistent storage management
• Troubleshooting cluster issues
• Network policies
• Kubernetes and Azure Networking:
• Kubernetes networking concepts
• Services, Ingress, and Load Balancers
• DNS resolution
• Pod-to-Pod communication
• Pod-to-Service communication
• Network Policies
• Service Mesh concepts, preferably Istio
• CNI Architecture
• Azure Networking:
• Virtual Networks
• Subnets
• Private Endpoints
• Peering
• Network Security Groups
• User Defined Routes
• Azure Firewall
• VPN connectivity
• Docker and Containerization:
• Docker image creation and optimization
• Container lifecycle management
• Registry management
• Security scanning
• Multi-stage builds
• Container troubleshooting
• Azure Container Registry
• Image versioning and governance
• Azure DevOps and CI/CD:
• Azure DevOps Pipelines
• YAML pipelines
• Build and Release pipelines
• Git repositories
• Branching strategies
• Infrastructure Deployment:
• Infrastructure as Code
• ARM Templates
• Bicep
• Terraform
• Automation:
• Deployment automation
• Environment promotion
• Release management
• Monitoring and Observability:
• Azure Monitor:
• Monitoring strategy implementation
• Alert configuration
• Dashboard creation
• Log Analytics:
• KQL query development
• Performance analysis
• Troubleshooting
• Application Monitoring:
• Azure Application Insights
• Distributed tracing
• Metrics collection
• AKS Monitoring:
• Cluster health monitoring
• Pod performance monitoring
• Resource optimization
• Azure Event Hub:
• Event Hub architecture
• Namespace management
• Consumer groups
• Scaling and throughput units
• Monitoring and troubleshooting
• Security and access control
• Azure Functions:
• Deployment and operations support
• Function App management
• Scaling configuration
• Managed Identity integration
• Monitoring and troubleshooting
• CI/CD deployment
• Apache NiFi:
• NiFi cluster setup and maintenance
• Flow deployment
• Security configuration
• Performance tuning
• Monitoring
• Troubleshooting
• Infrastructure as Code Must-Have Skills:
• Terraform
• Bicep
• ARM Templates

Benefits

• Platform provisioning
• Environment consistency
• Version-controlled infrastructure