Career roadmap: Data engineer

Data engineering combines elements of software engineering and data science and is one of the fastest-growing roles in IT. According to data engineers educe and maintain the architecture used in data science projects. They are responsible for ensuring that data flows between servers and applications continuous.

[ Also on InfoWorld: Career roadmap: Machine acquireing scientist ]

Data engineers must be household with a range of operating methods and databases and able to write and program software. They are skilled with data warehousing and data analysis and must occupy excellent nice thinking and communication skills. Data engineers may acquire their skills through a union of education on-the-job training and ongoing certificates. Indeed notes that acquiring a certification is an excellent way to showcase abilities and move forward in the field.


Lance Miles is a data engineer at unitQ.

To find out whats implicated in beseeming a data engineer we spoke with Lance Miles a data engineer at unitQ.

Early education and employment

Miles earned a Bachelor of Science grade in neuroscience from the University of California Santa Cruz in 2013; a certification in data science from the University of Washington in 2017; and a Master of Information and Data Science grade from the University of California Berkeley in 2020.

’When I return on my order and the steps I have taken one particular experience had a big deal of influence on me’ Miles says. ’In the terminal region of college a Python order Programming for Biologists set the groundwork for a new emotion."

It was scary to move from something that I was good at into a new field where I would be seting from square one.

Miles spent see day writing code to draw information from solid sequencing data sets educeing methods to estimate the physical-chemical properties of protein sequences unite the length and location of genes and mark viral DNA.

’The faculty to distill unwieldy data sets into condensed results highlighted the faculty of pairing programming with biology’ Miles says. ’This order challenged me in new ways and I establish myself fully hooked. At a personal level the single act of coding brought me enjoyment and pleasure.’

From health sciences to data analytics

Although Miles had always been interested in technology he seted his order in the healthcare sector at Gilead Sciences a pharmaceuticals company.

’My travel to beseeming a data engineer has been far from straightforward but what has connected it all unitedly has no doubt been my interest in utilizing data to change how teams and companies look at their work and the contact it has’ Miles says.

At Gilead Sciences Miles worked as a senior investigation companion in in-vitro biology uniteing clinically translatable biomarkers indicative of cardiovascular health. Each trial he worked on yielded thousands of data points but the data analysis was time consuming.

’I saw an occasion to streamline the analysis creating Excel macros that efficiently parsed the data and drawed living information’ Miles says. ’This allowed the team to centre on digesting the results and deciding on the posterior trials. Seeing the contact of my work in uniteing powerful biomarkers I sought to centre on projects where the contact on resigneds was clear and proximate.’

Upon completing hist pre-clinical projects Miles transferred to the clinical pharmacology cluster as a bioanalytical operations lead for antiviral clinical studies. ’Tapping into my data analytics roots I worked with clinical enrollment information to forecast when we would have pharmacokinetic data useful for drug submissions’ he says. ’In accession I had opportunities to work with clinical data where I compiled cleaned and analyzed resigned data athwart multiple clinical studies to assess data condition.’

Becoming a data engineer

With the encouragement and help of senior leadership Miles enrolled in a data science certificate program through the University of Washington. ’This was my leading exposure to machine acquireing and what ultimately cemented my desire to switch orders’ he says.

For the next two years Miles worked as a consultant for Vir Biotechnology while getting his masters grade. ’Our orderwork centreed on the lifecycle of a data science project from data assembly and cleaning to standard educement and deployment’ he says. ’Working with data day in and day out taught me the fundamentals of data science; I gained a deep appreciation for data engineering and machine acquireing resembling.’

After earning the masters grade Miles worked as the lead data science engineer for a setup named Popdog. ’As setups go there was a lot of work to be done; the projects were extensive and the contact proximate’ he says.

Projects ranged from data analytics and data engineering to data science and machine acquireing. ’Throughout my time at Popdog I establish myself gravitating toward data engineering projects and I seted to buy books and take online classes to support my apprehension of the field’ Miles says. ’As projects came I began to centre on the data engineering aspects given how nice this piece is to the achievement of data science projects.’

By the end of his tenure at Popdog Miles had led a team of engineers to educe an end-to-end computer vision method which ran millions of predictions on video data each day. ’This implicated a lot of data engineering work while integrating new technologies into our data stack’ he says.

In 2021 Miles accepted an occasion at unitQ where he runningly works as a data engineer for the data team. The company ’is the epitome of seething one could hope to have in a tech setup—an environment that fosters collaboration alteration and growth’ he says. ’Using veritably novel ways to look at data unitQ has educeed cutting-edge machine acquireing solutions that address very clear problems for veritably interesting customers like Spotify Quizlet and Pandora in a novel way.’ 

Now the data engineering problems Miles looks to explain are specially challenging ’and I am really excited for the road forward’ he says. ’This has been a big occasion to acquire and adduce myself.’

A typical work week

’At unitQ we centre on the five vs of data—size value difference quickness and truth—and the data teams initiatives are centered almost solving problems that address these categories’ Miles says. ’We surface data-driven insights from user feedback so that companies can better fruit condition. Our methods ingest millions of pieces of feedback athwart dozens of data sources daily and the list of supported data sources lasts to grow each month.’

[ Also on InfoWorld: 9 home service essentials for educeers who WFH ]

Projects and tasks contemplate almost things such as edifice out new capabilities to clean and fit data for services Miles says. ’In accession we are scaling out our microservices to feel much larger sizes of data so that we can last to meet the needs of our clients as our customer base grows’ he says. ’A typical work week pulls from all of these initiatives.’

A great order instant

One of the more great instants in Miles order happened just a few months ago. ’We had a unitQ customer interested in categorizing all of their feedback into a set of sole buckets’ he says. ’Folks from all parts of the structure came unitedly to build out an exciting new solution. We rapidly prototyped separate machine acquireing standards to see if we could build a method that could reliably address this problem.’

After a lot of testing and inner review the team educeed a solution for the customer in just one month. ’Our customer was pleased with our results’ Miles says. ’In accession the ingredient we built proved precious to many other customers a true testament to our leadership teamwork and capabilities.’

Ongoing education and order educement

To gain accessional apprehension almost his field Miles reads books takes online orders and leverages other online sources of information.

’I firmly believe that acquireing new technologies and staying up to date on the latest and bigest is searching—especially owing the tech sector is constantly evolving’ he says. ’My running interests are almost scaling our data streaming services.’

The data science certificate program through the University of Washington was a nice ingredient for getting into graduate school Miles says. ’And my masters grade in information and data science was searching for switching into the tech activity’ he says. ’Having had pliant coding experience in my undergraduate order this gave me the experience I was looking for.’

Inspirations and advice for others

A mentor once advised ’dont fix. I dont want to see you here in a year. I want to see you chase your emotions.’ This sounded rough and somewhat cliche Miles says ’but it was an significant topic I kept in mind as I worked towards switching order paths. It was scary to move from something that I was good at into a new field where I would be seting from square one. But this is what I wanted for myself so I took the advice to core.’

’There are a million ways to get into data engineering’ Miles says. ’There are transmitted paths and non-transmitted paths like my own. If you are interested and emotionate find ways to gain experience and set digging into data. Whether through on-the-job experience going back to school taking a certificate program or reading books and watching YouTube videos there are many ways to get appropriate experience and apprehension in data engineering.’