- Data Engineers are responsible for the creation and maintenance of analytics infrastructure, providing the foundations for data needs.
- Data Engineers are responsible for developing, constructing, maintenance and testing of architectures, such as databases and large-scale processing systems, implementing data flows to connect operational systems, and writing and executing ETL scripts and codes.
Summary of the Role and key responsibilities
- As a Data Engineer, you are responsible for developing accurate, efficient data solutions which meet customer needs to agreed timescales.
- You ensure the stability, robustness and resilience of the products you design and build and are in a position to effect changes to those products where necessary.
- As a Data Engineer you support continuous improvement of standards and provide leadership to develop Associate Data Engineers, providing technical guidance alongside other data engineering functions for customers.
- Translate business requirements to ensure they are clear and robust and can be transformed into reusable production ready code and / or effective data models.
- Define version control mechanisms and strategies for the source code of products.
- Support Quality Assurance colleagues and Data Managers responsible for the day-to-day testing and live running of data products and services.
- Shape solutions in a fit for purpose way; following the agreed principles and contribute to the overall Data Engineer Development lifecycle.
- Perform data profiling and quality measurements.
- Designing and developing reusable metadata libraries.
- Owning the stability of new products designed, including the on-going robustness, resilience, and stability of these products.
- Identifying, managing, and resolving issues preventing the delivery or continuous development of products.
- Supporting the growth of the team, by keeping abreast of market and industry trends, and sharing knowledge and experience with the rest of the team.
Programming & build
- Designs, develops. codes, tests, corrects, and documents simple programs or scripts under the direction of others.
- Understands core technical concepts related to their role and is able to apply them.
- Encourage joined up teamwork within own team and across other groups. Establish professional relationships with a range of stakeholders.
- Collaborate with these to share information, resources, and support. Invest time to develop a common focus and genuine positive team spirit where colleagues feel valued and respect one another.
Making Effective Decisions
- Understand own level of responsibility and empower others to make decisions where appropriate.
- Analyse and use a range of relevant, credible information from internal and external sources to support decisions. Invite challenge and where appropriate involve others in decision making.
Knowledge and Experience required
- Experience of Informatica or comparable ETL software.
- Experience of one or more programming languages (SQL, PL\SQL, Python, Scala)/
- Experience of developing data solutions on Oracle, Cloudera, or Horton Works platforms.
- Experience of Erwin or comparable data modelling software.
- Experience of developing data solutions using cloud native services for collection, storage, processing, analysis, visualisation, and security of data.
Other (Nice to have)
- Experience of working for a government dept.
- AWS / Hybrid Cloud technologies would be advantageous.