Big Data and Apache Spark Developer - Assistant Vice President

The Applications Development Senior Programmer Analyst is an intermediate level position responsible for participation in the establishment and implementation of new or revised application systems and programs in coordination with the Technology team. The overall objective of this role is to contribute to applications systems analysis and programming activities. Key Responsibilities: Design, develop, test, and maintain data processing applications using Apache Spark. Build scalable ETL and data transformation pipelines for large datasets. Develop and support applications using Java or Python in a distributed data environment. Analyze requirements and collaborate with business analysts, QA teams, and stakeholders to deliver solutions. Troubleshoot production issues and optimize application performance. Participate in code reviews and follow engineering best practices. Contribute to application modernization, migration, and automation initiatives. Leverage AI-assisted development tools to improve productivity and quality. Required Skills: 8-10 years of software development experience. Strong hands-on experience with Apache Spark. Proficiency in either Java or Python. Candidates with primary expertise in Python should have a good understanding of Java fundamentals (OOP concepts, collections, exception handling, multithreading basics). Experience working with SQL and relational databases. Understanding of distributed computing concepts and big data processing. Strong analytical, problem-solving, and debugging skills. Preferred Skills: Experience with Databricks (highly desirable). Exposure to cloud platforms and modern data engineering practices. Experience with CI/CD pipelines, Git, and Agile development methodologies. Knowledge of regulatory reporting or financial services domain is a plus. AI Mindset & Engineering Excellence Demonstrated ability and willingness to adopt AI-assisted development tools (e.g., GitHub Copilot, Devin, Databricks Assistant, or similar) to improve productivity, code quality, and delivery efficiency. Ability to effectively leverage AI-generated recommendations for coding, testing, debugging, documentation, and analysis while maintaining accountability for final deliverables. Strong engineering judgment to critically review, validate, and optimize AI-generated code and solutions rather than relying solely on tool outputs. Capable of identifying potential functional, performance, security, and maintainability issues in AI-assisted implementations and making appropriate corrections. Passion for continuous learning and staying current with emerging AI technologies and software engineering best practices. Nice to Have: Experience with Spark performance tuning. Familiarity with data lake architectures. Experience leveraging AI tools such as Copilot or similar developer productivity solutions.