Yousef Eddin

Deriva is a renewable energy company that specializes in generating and selling power, specifically solar and wind. They manage the sites themselves and then either sell a site as a whole or sell it per megawatt. It's a real business built on top of physical assets producing electricity, which means the data behind those assets has to actually be trustworthy. That last part is where I came in.

On paper I was a data analyst. In practice I was doing data engineering, and I'm glad I was. I got a lot of hands-on time with Databricks, Git, and PySpark, the kind of tooling that's hard to appreciate from a classroom.

Databricks is an extremely common data platform where businesses host a front end for their databases, run transformations, and build pipelines. From my seat it was where all the engineering happened, and I picked up a set of best practices that stuck with me:

• The medallion structure. Raw data lands as bronze, gets transformed into silver, and ends up completely clean as gold. Each layer has a clear job.
• DevOps for data. Writing tests for your code and using Git for version control and CI/CD, not just shipping notebooks and hoping.
• Streaming over batching. Let data update as it changes instead of waiting a full day to reprocess everything at once.

My managers were genuinely great. Kurt is one of the chillest, most experienced people I've worked with, sincere and straight to the point. John is a laid-back guy who's big on hiking and growing things. Both of them are gifted, kind analysts, and having managers like that early in your career makes a real difference.

The team itself is small, only about twelve people in the whole GitHub org. And here's the honest part: corporate coding culture is real, and I learned that the hard way. I once opened a simple, well-intentioned pull request. It didn't land well; someone felt protective of their code, it escalated to the team lead, and my Git access got pulled. It was a big, uncomfortable stink for a couple of days. Then the lead apologized, we talked it through, and everything got sorted out. It stung, but it taught me something no course ever would.

To understand the project, you have to understand how Deriva stores its data. There's site-level data (the site as a whole) and unit-level data. Sites can have outages, where they can't produce power, and communication failures, where the site and the database stop talking to each other.

Deriva cannot just ignore these problems. Left unfixed, bad data slowly corrupts everything downstream, lending decisions, PPAs, the works. My job was to catch these imperfections programmatically before they poisoned anything.

I created a GitHub repository for version control and connected it to Databricks for CI/CD. Then I started coding. My first instinct was Pandas, but the volume made that a non-starter, roughly 18,850 data points every single day. So I switched to PySpark and wrote the logic for three data-quality checks: zero generation (outage), long curtailment, and flatlines (comms outage).

Once the checks were written, the rest was implementing the pipeline on the medallion pattern. I took the raw bronze data down to silver by transforming a handful of columns, dropping the separate year, month, hour, and minute columns and consolidating them, then applied the checks. Finally I surfaced the gold data in a single dashboard plus an auto-updating Excel workflow for the stakeholder (John).