Java Developer

●Led a team in developing a Big Data task alert system, reducing response time from 2 hours to within 30 minutes. ●Led a team in drafting an in-depth technical report on Flink and introduced Flink technology, resulting in a 42% improvement in real-time task development efficiency. ●Developed a Flink-based streaming platform using Dolphinscheduler for launching and monitoring Flink SQL tasks, and interfaced with international clients to assist in the implementation of an AI team's network fault intelligent diagnostic system.

●Integrated offline and real-time workflows, reducing the data ingestion time for MySQL data into the data lake from hours to minutes. ●Designed a real-time stream processing workflow using Flink to write data from Kafka Binlog to Hudi, reducing data update time from 3 hours to 18 minutes to address timeliness issues in the intelligent replenishment algorithm caused by data latency. ●Analyzed source code and found that data duplication occurred after initializing "index.bootstrap" with multi-parallelism and reported the bug to the GitHub community. ●Implemented a customized KeyBy function in Flink SQL to support the parallel processing of 1.2M data entries per second (peak 1.5M per second). ●Developed custom ClickHouse modules for enhanced parallel processing and increased write throughput. ●Designed event-tracking data flows for different data terminals to provide compatibility. ●Developed programs to write event tracking data into AWS S3 using Hudi to reduce the small files. ●Implemented data reading and writing into Kafka using Confluent Avro to reduce network bandwidth usage by 18%.

●Implemented communication between the Job Manager and Flink Agent to manage the job status. ●Developed Flink job submitter, supported yarn-per-job and standalone. ●Implemented Flink sql for Kafka single and double stream join. ●Developed a file monitoring program to auto-update Class Loaders upon changes in module file paths. ●Implemented job submission and monitoring communication between components such as Job Manager and Batch Executor.