Senior DataOps Engineer - MainApp infrastructure
Shenzhen, China / Shanghai / Hangzhou / Guangzhou / Chengdu Engineering – Mainapp / Full-time / Hybrid We are a team to design, develop, maintain, and improve software for various ventures projects, i.e., projects that are adjacent to our core businesses and are bootstrapped fast with a lean team. You will be actively involved in the design of various components behind scalable applications, from frontend UI to backend infrastructure.Responsibilities:
- Manage the daily operations of Hadoop, Spark, Flink, Clickhouse, and other big data platforms to ensure system stability and data security;
- Operate and administer Linux systems, proficient in common commands, with scripting skills in Shell, Python, or similar languages;
- Maintain and operate big data components on cloud platforms (AWS/GCP/Azure), with hands-on experience in container technologies such as Kubernetes and Docker;
- Build and maintain monitoring systems to track platform performance, optimize configurations and resource allocation, and troubleshoot performance issues (tools such as Datadog, Prometheus, Grafana, Zabbix, etc.);
- Deploy, upgrade, and scale core big data components including Flink, Kafka, Spark, Impala,Clickhouse and Kudu;
- Write and maintain technical documentation for operations, and support in platform solution design and upgrades;
- Explore and adopt new technologies to optimize operation processes and provide technical guidance to help client teams improve their capabilities.
- Preferred: Experience on the development of n8n workflow
Qualifications:
- Familiarity with CDH/CDP/HDP or similar Hadoop distributions, with proven experience in big data platform operations preferred;
- Proficiency in Linux administration and scripting (Shell, Python, or equivalent);
- Experience with major cloud platforms (AWS/GCP/Azure) and containerized environments (Kubernetes, Docker);
- Strong knowledge of big data components and performance tuning, able to resolve stability and scalability issues independently;
- Strong technical writing and communication skills to support documentation and solution delivery;
- Self-motivated with a passion for new technologies, able to apply innovative approaches to optimize operations and drive efficiency.