mark cerqueira well-rounded nerd

Managing Management - Interview Journey and Calibration

Instrumenting the journey of people you interview is critical to understanding and improving your interview processes. With instrumentation you set up concrete feedback loops that give you insight and opportunities to improve your interviewing. I’ve used two Applicant Tracking Systems (ATS), Lever and Greenhouse, and neither provided the data in the format I wanted. Spreadsheets are SaaS apps waiting to be built so let’s walk through the spreadsheet solution I built to instrument the interviewing funnel.

In this post we’ll walk through the spreadsheet I built with my fantastic recruiting partner Daren to help solve this problem: Candidate Journey Tracker. This sheet has sample data for interview loops that are simple: one phone screen and if the candidate advances we have an onsite consisting of three technical interviews. We use the Lever system for evaluating candidates: 1 (strong no hire), 2 (no hire), 3 (hire), 4 (strong hire).


Our first stop is the Candidates sheet. For any candidate that advances to the onsite stage we add their information to this sheet. This gives us a single place to review all candidates we’ve brought onsite in a single place and easily answer questions like:

  • What do onsite scores look like for offer-stage candidates today and in the past?
  • Are there any trends between candidate origin and onsite result?
  • How do candidates a particular person advances from phone screen do in onsites?

With candidate journey data collected in a single place we can now do interesting analysis around calibration. Calibration here means alignment between an each interviewer’s assessment of the candidate and the final decision for the candidate. An interview is more calibrated if they are inclined when the panel is inclined or when they are not inclined and the panel is not inclined. When there is a mismatch, the interviewer is less calibrated.

Calibration Analysis

For our sample data above you’ll notice that Day9 is perfectly calibrated: whenever Day9 is inclined to hire, the panel is inclined to hire and when he’s not inclined, the panel’s not inclined. Artosis is the exact opposite; some say he’s cursed. Tasteless is more a coin-flip: half the time he’s calibrated.


There’s no need to do this analysis manually though! Using Google App Scripts (Tools -> Script Editor) you can write custom functions that calculate calibration percentage for every interviewer. The Calibration Analysis sheet shows off the custom calibrationPercentage function.

Candidates Revisited

With this function we can revisit our Candidates sheet and do some math. Astute observers will notice the sheet had the average score of onsite interviews and also a calibrated average of onsite interviews. The calibrated average adjusts each interviewer’s score based on their calibration percentange. Unless you’re perfectly calibrated this means your scores will be adjusted downwards. For our sample data, Day9’s scores aren’t adjusted (e.g. a 3 stays a 3) because he is perfectly calibrated; Tasteless’s scores are calibrated downwards by 50% (e.g. a 3 becomes a 1.5) because Tasteless has a 50% calibration; and Artosis’ scores are ignored (e.g. a 3 becomes a 0) because he is never calibrated. This calibrated average score provides another lens through which to view a candidate’s performance.

There are a few things worth highlighting before we close out this post:

  • This spreadsheet should not be used for deciding whether to move to offer-stage with a candidate. Having past data and concrete numbers in front of you can tempt you into simply making the offer decision a numbers game but you should avoid doing this.
  • Our sample data has three technical interviews but real onsites are rarely so uniform. Different interviewers will probe different technical and behavioral competencies. If a particular competency is easily handled by many candidates or catches many candidates off guard this will affect that particular interviewer’s calibration.
  • Calibration percentage calculation looks at all data points for an interviewer and applies that percentage to all their scores. An alternate system would only look at past data points when calculating this percentage, but this would hurt data sample size for earlier interviews.

In closing, this spreadsheet has been helpful tool to help assess the quailty of our interviewing and there’s so much more that can be built off of it (e.g. percent accuracy for phone screeners). Got an interesting tip that helps you better understand how calibrated your interviewing is? Let me know on Twitter!

Managing Management is a series on systems that are shareable and serve as actionable templates for addressing the deluge of things managers encounter everyday. Enjoyed this post and want to see more? Check out more at Managing Management.

One Week to Xenoblade Chronicles: Definitive Edition

Xenoblade Chronicles: Definitive Edition is less than a week away from release on the Nintendo Switch and I am so excited! Times were dark for JRPGs in 2010 when Xenoblade Chronicles released on the Wii but the game revived faith in the genre for me and many others. Ten years later not only are we getting a visually spruced up version, we’re getting some remastered audio tracks, improved menus, a new epilogue - Futures Connected, and many quality-of-life improvements. This great article by Nintendo Enthusiast dives more into some of these changes.

To compound on this excitement, in a recent interview with Famitsu Xenoblade series director Tetsuya Takahashi confirmed there is a team at Monolith Soft working on a new game! High on my list of wants from Monolith Soft is a remastered version of Xenoblade Chronicles X with a new epilogue to properly close out that story, but I’m super excited to hear a “new game” is in the works. I hope it’s a new Xenoblade entry and has more Gundams!

If you’ve never played Xenoblade Chronicles before and have liked JRPGs at some point in your life, I cannot recommend this game enough! If you’re not sure about the genre and you’ve been meaning to check out a solid JRPG, this is a golden opportunity!

P.S. It’s Reyn time again!

Happy 7th Birthday to

As I approach the ten year anniversary of my move from New Jersey to California to begin my career, today marks seven years of blogging on!

My first post on Android Annotations, Ant, Dalvik Executable (dex) feels like forever ago. The Android ecosystem has evolved so much since then: Kotlin has become a first-class Android programming language, Apache Ant has been replaced by Gradle, Android Runtime (ART) has supplanted Dalvik, and most of the great stuff provided by Android Annotations is now built into the default Android development experience.

Here’s to hoping we can celebrate’s 8th birthday in a post-COVID-19 world. 🙏

Manager Reads - Team Topologies

Organizational design has always interested me and thanks to Spotify I finally got a book recommendation so without further ado… In this edition of Manager Reads we cover Team Topologies: Organizing Business and Technology Teams for Fast Flow by Matthew Skelton and Manuel Pais. This is also the first Manager Reads entry featuring a Portuguese author: Olá Manuel! 🇵🇹

Who recommended the book? Several months back I read the Spotify white paper on Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds. A recent blog post by Jeremiah Lee, who worked at Spotify in 2017, highlighted the failure of the Spotify organizational model: Failed #SquadGoals - Spotify doesn’t use “the Spotify model” and neither should you. In this post, Lee recommends reading Team Topologies for more information on the Spotify model. Lee also has great taste in books as his post includes a quote from one of my favorite Manager Reads books: Turn the Ship Around!

Judge the book by its cover, font, page quality? Team Topologies is a well-designed book inside and out. The cover features the shapes, lines, and colors used inside the book to describe different organizational models. The inside really shines too. The book is divided into three parts and each part has a designated color. The chapter markers along the edge of the pages, the occasional tips and notes highlighted, and chapter numbers all use this designated color. Everything is really cleanly laid out; you can tell the authors invested energy in making this book look great to get high marks in this Judge a Book by Its Cover section of Manager Reads! 😁

Thoughts on the book and the big take-aways? Team Topologies is broken up into three parts that cover:

  1. Teams As the Means of Delivery - This is the theory section of the book and covers things like Conway’s law which states that the architecture of solutions built will mirror the communication structures of organizations; cognitive load and the importance of keeping this load low; and how we should view teams, not individuals, as the most effective means of software delivery.
  2. Team Topologies that Work for Flow - This section covers the four fundamental team types: stream-aligned, platform, complicated subsystem, and enabling. The majority of teams in an organization should be stream-aligned and the focus of other teams should be on empowering stream-aligned teams. A stream-aligned team focuses on a continous flow (the stream in stream-aligned is to emphasize the importance of flow) of work that is aligned to a business domain. When considering domains for teams this section also covers fracture planes; these are natural seams in software systems that allow splitting into logical subcomponents that can be owned by a single team.
  3. Evolving Team Inreactions for Innovation and Rapidly Delivery - This last section covers the three types of interactions the four fundamental team types can have with each other: collaboration, X-as-a-Service, and facilitating. Depending on the situation different interactions may work best as there are advantages and disadvantages to the three interactions.

Four team types and three interaction types sounds small but thinking about Twitch’s organizational design most teams and their interactions with other teams fit somewhat cleanly into this model. I can also attest to the advantages and disadvantages of the interactions having seen and experienced them myself!

The one thing I wish Team Topologies would provide is a longer-lived case study for an organization. There are smaller case studies sprinkled throughout the book but walking away from the book none of them really stuck with me because they left a very shallow impression. It would’ve been great to have a recurring case study focused on a single organization that was revisited throughout the book as we learned new concepts and principles.

Favorite quote from the book? “‘Minimize cognitive load for others’ is one of the most useful heuristics for good software development.”

Elevator pitch for suggesting (or not suggesting) the book? Team Topologies serves as both a primer to organizational design and presentation of what it believes is a good organizational model to optimize for getting things done. It’s a nice mix for those looking for a mix of theory and praactical applications of organizational design. As far as this is the first book I’ve read on organizational design, I liked it and recommend it.

Notes in Evernote? Check out my notes on 🐘.

Most of the world’s wisdom is written down in its best books. Manager Reads is a series covering books on management and leadership, focusing on books that can improve your own leadership with the wisdom of others. Enjoyed this post and want to see more? Check out more at Manager Reads.

Xcode is Not Environmentally Friendly

Dread sets in whenever a new version of Xcode releases. It’s dreadful because Xcode is a bloated monolith; downloading Xcode means downloading a 7.5 GB bundle that includes an IDE, simulators, and the appleOS family of SDKs, all from scratch, every single time there is an update. Frustration and dread is compounded with frequent failed downloads from Apple’s Developer website and the random, silent failures when downloading from the Mac App Store.

Compare this to Android where the IDE, Android Studio, is a comparatively light 768 MB download. Updates to Android Studio are downloaded and applied as small patches. The Android SDK, build tools, and emulators live separately from the IDE so when you get a new version of Android Studio there’s no need to download all that stuff again.

Ultimately Xcode is not environmentally friendly. From the article Does Irresponsible Web Development Contribute to Global Warming? by Emerge Interactive:

According to the American Council for an Energy-Efficient Economy it takes 5.12 kWh of electricity per gigabyte of transferred data. And according to the Department of Energy the average US power plant expends 600 grams of carbon dioxide for every kWh generated. That means that transferring 1GB of data produces 3kg of CO2.

A download of Xcode generates 22 kg of carbon dioxide. That’s the equivalent of burning two and half gallons of gasoline! A more modularized Xcode would not only be more environmentally friendly, it would make updates for developers less dreadful. Be green and be kind, Apple!