frankxieke

1. Bigdata warehouse construction. For example, ETL work from HIVE to Doris sync to ensure query efficiency. In order to improve query efficiency for BI usages. Build a ETL service, which scan all tables in hive, detect detect hive schema and build table in OLAP database(DORIS). Support Full Sync for small table and Incremental sync for partitioned hive table according its partitioned field. Build data warehouse for company, especially for ODS layer which sync from original system, and build wide-table for DWD with multiple dimensions. Using Doris as storage layer and open source BI (DataEase) to explore data and build dashboard.

Open Source Message queue pulsar/bookkeeper

Build Distributed file system like open source cubefs which can be mounted on container using fuse. With In-memory metadata server which stores name,directory. Data-nodes store partitioned file and location is stored on metadata server.

For example log data collector to Kafka, Kafka to HDFS dumper, Kafka Mirror between clusters. Build distributed Kafka sink to HDFS service as KafkaStreamDumper which uses nosql to store message committed partition offsets and master node splits partitions among workers and different work consume messages and write into HDFS separately.

Build distributed Nosql platform which uses rocksdb as storage layer with stateless proxy to handle client requests. Provide users with different programming SDK. Build distributed Object Storage with restful interface. The principle is like HayStack from facebook. The file uploaded will be split as small needles with multiple replicas and a lot of needles stored in datanode as bigfile. The metadata of the file (name,location,bucket) is stored in Nosql database.

Mainly responsible for the development and maintenance of CDN media servers for content distribution networks．