Python ETL Developer
Python ETL Developer
IDPS (Index Data Processing Platform) is a program whose main purpose is to drive improvements across data collection, data management and data governance resulting in improved efficiencies, robustness, and the ability for our business to use our data as an asset by driving both cost savings and revenue opportunities. Some of the outcomes and benefits of this program include:
- Provides a single, centralized version of the truth/data including standardized logical and physical data model
- Data capabilities to build world class multi-asset indices and data & analytics services
- Universal data quality profiling and checks as well as Data Policies and Governance
- A platform for data discovery, management and cataloguing
- Ability to on-board and integrate new business acquisitions more efficiently
- Improve operational efficiency through automation
- A customized bi-directional packaging and delivery capability
- We are looking to fill a Python ETL Developer who will play an integral role in the ongoing application & data analysis, design, development, integration, enhancements and maintenance of the data platform which will span a broad range of data and functionality. While constantly driving innovation and working on complex technical challenges every day, this team member will work closely with BAs, PO's, PMs and other team members to understand business requirements, design and develop quality technical solutions that are aligned with the business and platform strategies while ensuring compliance with the organization's architectural and coding standards. In this role, the developer will also be responsible for helping to troubleshoot complex support issues on an as needed basis. This role will be based in Malabe - Sri Lanka, Bucharest - Romania on the MIT - Shared Tech Team.
- Work together with other developers, business analysts and product owners to ensure that new or modified ETL applications are developed using industry standard best practice design and development practices
- Contribute to the development, delivery and maintenance of ETL solutions
- Design, prototype and develop optimal code to maintain exceptional performance in processing large volumes of data across diverse sources
- Build solutions using SAP HANA and/or AWS Cloud native ecosystem and components
- Design, develop and test data models and ensure that data is correctly loaded into the system
- Application performance analysis and performance optimization
- Defining and maintaining robust and scalable ETL to manage data according to formal concepts, principles, and techniques
- Design and develop scalable, reusable ETL code in line with architectural standards and best practices
- Make recommendations for data engineering related functions that result in a more cost-effective product delivery
- Help to manage data as a critical asset within IDPS business functions
- 3+ years hands on experience in Python based ETL Application development experience
- Experience in executing/monitoring ETL jobs/pipelines and data engineering support on production environment
- Sound knowledge & hands on experience in troubleshooting and performance optimization of Python ETL applications and SQL code.
- Hands on experience in developing ETL jobs, workflows & data-pipelines.
- Experience with Python frameworks such as Bonobo, pygrametl is a plus
- Experience in data transformation features such as: Preserving, Hierarchy Flattening, Match Transform, Data Cleansing Generator, Lookup, Fuzzy Lookup, etc.
- Hands on experience implementing SCD Type 2 dimensions, to maintain the history of dimensional data
- Experience in Data Warehousing or Data Lakes which includes Data integration concepts, analysis and development of Dataflows, mappings using needed transformations
- Experience with end-to-end large Data Conversion Projects to migrate large volumes of data from SQL DBs, NoSQL DBs, File sources, etc. to a target database/warehouse.
- Exposure on open-source big data stacks (preferred kudu/impala/hive/kafka/flink)
- Experience in Cloud Native Big Data stack & Data Warehousing implementations such as Snowflake is a plus.
- Experience with Financial Markets domain is a plus
- Experience in BI tools is a plus
- Experience using SAP ETL Tools such as Data Services and Data Intelligence is a plus
- SAP Experience with SLT, SDI a plus
- Knowledge of SAP HANA Advanced Modelling (SAP HANA Text, SAP HANA Graph, SAP HANA Spatial, SAP HANA PAL) a plus
- Experience Developing SAP HANA XS/XSA applications based on the HANA architecture a plus
- Bachelor's Degree in MIS, Computer Science, or IT related field or equivalent IT related experience
- Solid understanding of databases and SQL/Analytical SQLHands on experience working with the IDE's like Zeppelin, Jupyter, Notebook, PyCharm etc
- Hands on experience in using build/deployment tools such as PyBuilder, Pip, Jenkins, Docker
- Hands on experience in using Pandas, PySpark, Kafka, Boto3, etc.
- Hands on experience on ETL orchestration tools - preferably SAP HANA, Nifi, Airflow, AWS Glue, AWS DMA
- Fundamental understanding of ETL stages and techniques
- Demonstrated knowledge & experience in OOP/OOD
- Demonstrated ability to understand business requirements and the implications of those requirements on current and future roadmaps
- Understanding, at a detailed level, of all architectural dependencies of technologies in IT environments
- Self-motivated and proactive with demonstrated critical thinking and creative problem-solving skills
- Agile development processes (SCRUM and Kanban)
- Strong communication, presentation, documentation and interpersonal skills
- Able to self-manage and work independently in a fast-paced environment with dynamic requirements and priorities
- Experience with Jira a plus
Diversity & Inclusion
People are at the heart of what we do and drive the success of our business. Our colleagues thrive personally and professionally through our shared values of Integrity, Partnership, Innovation and Excellence are at the core of our culture. We embrace diversity and actively seek to attract people with unique backgrounds and perspectives. We are always looking at ways to become more agile so we meet the needs of our teams and customers. We believe that an inclusive collaborative workplace is pivotal to our success and supports the potential and growth of all colleagues at LSEG.About Us
London Stock Exchange Group (LSE.L) is a diversified international market infrastructure and capital markets business sitting at the heart of the world's financial community. The Group can trace its history back to 1698.
The Group operates a broad range of international equity, bond and derivatives markets, including London Stock Exchange; Borsa Italiana; MTS, Europe's leading fixed income market; and Turquoise, a pan-European equities MTF. It is also home to one of the world's leading growth markets for SMEs, AIM. Through its platforms, the Group offers international business and investors unrivalled access to Europe's capital markets.
Post trade and risk management services are a significant part of the Group's business operations. In addition to majority ownership of multi-asset global CCP operator, LCH Group, LSEG operates CC&G, the Italian clearing house; Monte Titoli, the T2S-ready European settlement business; and globeSettle, the Group's newly established CSD based in Luxembourg.
The Group is a global leader in indexing and analytic solutions. FTSE Russell offers thousands of indexes that measure and benchmark markets around the world. The Group also provides customers with an extensive range of real time and reference data products, including SEDOL, UnaVista, and RNS.
London Stock Exchange Group is a leading developer of high performance trading platforms and capital markets software for customers around the world. In addition to the Group's own markets, over 35 other organisations and exchanges use the Group's MillenniumIT trading, surveillance and post trade technology.
Headquartered in London, with significant operations in North America, Italy, France and Sri Lanka, the Group employs approximately 4,700 people.Values & Behaviours
Integrity: My word is my bond. Integrity underpins all that we do - from unshakable commitment to building and supporting global markets based on transparency and trust, to every transaction across our business with each and every stakeholder. We are a source of enduring confidence in the financial system, so when we say that our work is our bond - we mean it.
Partnership: We collaborate to succeed. We pride ourselves on working together as proactive partners, building positive relationships with our colleagues, customers, investors, regulators, governments and shareholders - for our mutual success and the benefit of all.
Innovation: We nurture new ideas. We are ambitious and forward-looking - a pioneering Group of market innovators, driven by fresh thinking that has kept us ahead of change. We prudently and proactively invest to make sure that out markets and services constantly moving forward, developing and evolving with advances in technology.
Excellence: We are committed to quality. We have a fundamental commitment to developing talented teams who deliver to the highest standards in all that we do. By collaborating together, we will sustain industry-leading levels of excellence, setting the benchmarks that inspire ever better performance.
We are looking for intellectually curious people, passionate about the bigger picture of how technology industry is evolving, ready to ask difficult questions and deal with complicated scenarios. If you are creative and a problem solver, this is the place to be as will be supporting you to fast-forward your career.
We maximise each employee's potential through personal development and through a wide range of learning tools both formal and informal. We believe that remunerating our people with competitive salaries and a range of attractive benefits such as:
- 25 days' holiday per year
- 1 additional day off for your Birthday
- Annual wellness allowance
- Share Purchase Plan
- Medical Health Insurance
- Medical Subscription (preventive medical services)
- Dental Coverage
- Life insurance
- Pillar 3 Private Pension
- Meal Vouchers
- Flexible Benefits
- Volunteering Events
- Team Buildings
At LSEG, we believe that creating a diverse and inclusive organisation is fundamental to the way we deliver on our promise - and our purpose. You'll be part of an organisation of over 25,000 people, spanning 70 countries. We recognise the individual perspectives each of our colleagues brings, and our diverse workforce is one of our greatest strengths. In supporting collaboration and creativity and encouraging new ideas across a diverse and inclusive workforce, we can improve how we are driving financial stability, empowering economies and enabling customers to create sustainable growth.
LSEG offers a range of tailored benefits and support from healthcare and retirement planning to paid volunteering days and wellbeing initiatives.
We are an equal opportunities employer. We do not discriminate on the basis of race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or any other basis protected under applicable law. In accordance with applicable law, we make reasonable accommodations for applicants' and employees' religious practices and beliefs, as well as any mental health or physical disability needs.
Please take a moment to read this privacy notice carefully, as it describes what personal information the London Stock Exchange Group (LSEG) ("We") may hold about you, what it's used for, and how it's obtained, your rights and how to contact us as a data subject.
If you are submitting as a Recruitment Agency Partner, it is essential and your responsibility to ensure that candidates applying to LSEG are aware of this privacy notice.