Big Data Developer

Design, develop, and implement highly scalable distributed big data solutions using Hadoop ecosystem technologies including HBase, Hive, Kudu, and Spark. Architect HBase schemas and data models to accommodate evolving business requirements while optimizing performance for storage and retrieval. Develop complex Hive queries and data processing pipelines to transform raw data into structured formats, and implement real-time ingestion pipelines using Spark Streaming and Spark SQL. Optimize Spark applications for performance through RDD tuning, data partitioning strategies, and in-memory caching. Utilize Spark MLlib for machine learning tasks such as classification, regression, clustering, and collaborative filtering. Collaborate with data scientists to integrate models into production workflows for real-time predictions. Troubleshoot performance bottlenecks, data quality issues, and system failures in big data applications. Stay abreast of emerging technologies and best practices to evaluate their impact on architecture and solutions.