KPI's of an IT Development Project

Measuring KPI's or Key Performance Indicators is a way to check health of a project. As the name suggests, these indicate the performance of a project.

It is essential to define the KPI's at the start of the project, in fact at the proposal stage. These can vary depending on the type of project (support, development, testing). In this article, I will be focusing on Development project KPI's. These are just a set of guidelines and you can change, add and modify these based on the requirement of your specific project.

KPI's can typically be divided into 5 broad groups: Schedule, Productivity, Quality, Governance and Customer Experience

Schedule KPI's

---

Number of Releases Agreed vs Actual

Defines the number of development releases a team shall do in a specified time period. 
e.g. 1 release per Quarter

Schedule Variance

One of the most common ways to measure the schedule it to calculate how early or late release was delivered as compared to the baseline date. 

Productivity KPI's

---

Number of Stories Committed vs Delivered

In case of a Agile project, it defines the number of stories that were committed at the start of a Sprint vs actual number of stories that were delivered. Another variation of this which makes slightly more sense is 

Number of Story Points Committed vs Delivered

e.g. 20 story points per Sprint. Similar KPI can be defined and measured at a Release level (Release comprises of many Sprints)

Quality KPI's

---

Defect Density

Number of weighted defects per story point

Defects can be weighted as per the Priority of the defect. e.g. High priority defect given a weight-age of 10, Medium priority 5 and Low Priority 1. In case of 1 High Priority and 3 Low priority defects, total weighted defects can be counted as 1 x 10 + 3 x 1 = 13 weighted defects.

This KPI can be measured in Development as well as UAT phase of the project.

DDE (Defect Detection Efficiency)

This KPI's measures the the effectiveness of the testing team to detect defects before it goes to next phase of project life cycle, typically from  Development phase to UAT phase and a good way to judge the teams QA effectiveness. It is defined as 

Total Number of system test defects / (Total Number of system test defects + Total Number of UAT defects) * 100

If required this KPI can be compared as per the priority of the defects to give a fair comparison of pre and post UAT defects.

Reopened Defects

A good way to measure the quality of fixes in a phase is the measure how many of the defects are being reopened. This is measured as 

Number of Defects reopened / Total number of UAT defects

This KPI can be measured in Development as well as UAT phase of the project.

Test  Coverage

Test coverage is a good indication of how good testing team is thinking of various testing scenarios. Better test coverage shall result in more DDE.

Number of functional test cases per Story Point

Code Coverage

Writing unit test cases for the code is a good way to prevent the kind of defects which are difficult to capture as part of functional testing.

Number of Unit Test Cases per 1000 Lines of Code

Story Acceptance

Story acceptance by Product Owner is a great way to test if development has been done by team as per story specifications

Number of stories Accepted/Number of stories Delivered

Governance KPI's

---

Often ignored, but one of the most important set of KPI's are Governance KPI's. Why ? Lets see..

Weekly meetings planned vs Actual 

Communication with customer both formally as well as informally is the need of the hour as more often than not communication failure is a reason for project failures. 

Number of weekly meetings planned /Number of actual meetings 

This KPI is a great way to measure customer engagement and same can be measured for monthly Governance meeting or Quarterly Business meetings which are more strategic in nature

Customer Experience KPI's

---

Customer Satisfaction Survey (CSS)

Taking formal feedback from the end customer is the ultimate way to gauge the quality of services provided. This shall be taken at least quarterly and can be measured on a scale of 1 to 5.

Number of escalations in a quarter

Number of formal escalations by customer per quarter is measured as a to gauge the Voice of Customer.

The above list is only a sample of KPI's to get you started. You can think of many more that suits your project needs. While it is always a good idea to define and measure KPI's in such a way that it reflects the true health o f the project, but let me warn you with a word of caution. Do not get blind folded only by the numbers KPI dashboards give !!! Though formal customer communication is showcased as one of the KPI's above, but nothing can beat a casual talk with customer, so just pick up the phone can call him/her now:-)

--------------------X--------------------

Fixed Price Projects - My cup of Tea ?

"I don't have good Gross Margins for my project"

"Customer keeps on changing the requirement"

"This requirement was not written in the document"

"Sales team undersold this project"

"There are contractual flaws leading to less margins"

Sounds familiar ?

Today IT landscape is changing drastically and so is the customer. Customer want to do more and more projects in a fixed price fashion. There are majorly 2 reasons of this:

• Safeguard the budget
• Not enough trust built between customer and vendor (specially in case of new projects)

So let me come to the point. In this article I will try to recommend few ways that can help to answer few of the above questions so that it is a WIN WIN for both customer (gets what he wants) and vendor (delivers with good gross margins). First thing first. Half of the battle is either won or lost in presales phase. Let's talk about it a bit.

Pre-sales

---

Requirements

• If requirements are not elaborate, ask questions (lot of questions), educate customer to help him include implicit requirements (e.g. sorting, pagination, filtering etc. in case of a UI requirement)
  
• Invest time of a Business Analyst to refine customer requirements. Believe me this investment will go a long way both to elaborate requirements and even avoiding ugly clashes later in the project (on implicit vs explicit requirements)
• Always remember most of the time customer will provide a Business Requirement Document (WHAT he wants) and estimation should not be done without converting it into Functional Specification or a detailed Use Case Document (HOW and a flow of use cases)
• If there is a end to end product that needs to be build, it is always good to have a Minimum Viable Product (MVP), which can be a Fixed Price. In this phase a good rapport can be built post which rest of the product can be built in a T&M mode.

Contract

• I will recommend for a Business Analyst in a project to be billed who will be team's face for discussing any requirements with customer else we end up taking a risk of Change Requests leaking into the team (customer talking directly to developers is a very common case) without a Scrum Master aware of it.
• Better to add a contingency in the overall effort to safeguard the situations where unexpected requirement (implicit requirement) that is not planned for
• If part of the team is expected to work in customer timezone, same have to taken care in terms of cost of the resource
• When few projects are already under execution phase, we are tempted to assume that we will use an existing Project Manager and hence do not plan for his/her cost in initial deal. What if your new project gets signed off a bit late and by that time your previous project has ended? I guess you know what to do. So better to plan for this kind of a situation.
• Acceptance criteria is normally not defined well and invoice milestones are linked to Acceptance criteria. e.g. "UAT sign off" is stated as one of the acceptance criteria. It should be as quantitative as possible like "No P1/P2 open defects"
• Cost of delay: This probably is a very important clause that can help all teams to make sure they are delivering on time. 

Development

---

• Try to on board the team in a staggered manner to avoid a lot of cost upfront. This also helps for Senior members to define some ground rules. Getting too many people at the same time can be counter productive.
• Add Change Request process to be part of kick off presentation, so that the definition of Change Request is mutually agreed between all parties to avoid conflicts later.
• Avoid Change Request leakage. In case of a change request be as objective as possible and have an open discussion with customer.
• Most common and most difficult way to control cost is by doing Pyramid correction. Try to have a good mix of team experience
• Try to cross train team members
• In case teams are working in shifts and are coming in company cabs, make sure to cancel cabs in case someone is on leave. This is a small amount but still impacts your bottom line

Hopefully this will help get answers to some of the points we discussed at the start of this article. Please feel free to share your experiences.

--------------------X--------------------

Priority vs Severity vs Impact

I have come across multiple project managers and teams managing Level 2 and Level 3 IT support projects who struggle to articulate the EXACT definitions of Priority, Severity and Impact. Also these 3 terms (especially Priority and Severity) are used interchangeably by project teams. Here is my small attempt to explain the same with some examples.

Impact is the way in which an Incident affects the core business or revenue of the organization. A high impact defect has the potential to stop core business operations or even cause revenue losses. Typically Impact of the Incident does not change with time. Below you can see sample definition of various levels of impact. These can be changed or modified as per the business you or your team is supporting.

1-High: Defects resulting in failure of key Organization business processes impacting revenue. There is no work around available.

2-Medium: Defects impacting business processes. Key components are down and major functions are not working. No immediate / serious impact but has potential to affect revenues. Unreasonable work around exists.

3-Low: Defects impacting functions that have a low/negligible impact on key business process. Reasonable workaround exists.

Urgency suggests the amount of delay acceptable in resolving a particular incident and is always associated with time. A high urgency incident will have minimal acceptable delay in resolution. The Urgency of the Incident can increase as the age of the Incident increases.

1-High: Very urgent incident which should be treated as quickly as possible

2-Medium: Urgent incident which should be treated quickly

3-Low: Incident that is not urgent and resolution can wait for some time

Priority is defined as a combination of Impact and Urgency (Priority=Impact X Urgency) and is defined as per the below table

	Impact
Urgency		1-High	2-Medium	3-Low
	1-High	P1	P2	P3
	2-Medium	P2	P3	P4
	3-Low	P3	P4	P5

Below example will help you to understand the above definitions better 

Example: Incident logged that a particular feature has to be added on www.phutwa.com site effective 31-Dec which will increase user hit rate on website. For this to happen QA team need to provide sign off on this feature in lower environment by 01-Dec. Development team started working on the feature, but while testing it in lower environments team found it had some defects.

• 01-Nov: On 01-Nov, the defect will have Impact: Medium, Urgency: Low
• 20-Nov: If still unresolved, Impact:Medium, Urgency will change to  Medium
• 30-Nov: If still unresolved, Impact:Medium, Urgency will change to High

In the above example Urgency of the defect remains same but its Urgency keeps on increases as we reach near to a critical date of 01-Dec.

Hope the above definitions and examples will clarify the differences and relations between all 3 terms. Please feel free to comment with your views.

--------------X--------------

My team is very productive. Really !!!

In today’s competitive world, majority of the organizations (IT service Industry in particular) are facing a challenge to sustain good gross margins. There are many proven as well as new and innovative ways that have been devised by these organizations, that will help them improve on gross margins. Top most of these being improving the productivity of the existing team.

I am going to talk about only one way which has been implemented in our company and holds the potential to improve productivity.

Typically there are 2 types of people in the project (Makers and Managers). Makers are primarily developers and testers whose main focus is on designing, coding, testing, development, creative work etc. and Managers are (as name suggests) whose main work is planning, preparing for meetings, doing meetings etc.

Makers are the people who need Sustained Attention (i.e. manage high attention spans as they need to be focused for longer duration in order to write good code, do good testing etc.), while Managers need Alternating Attention (as they need to focus on multiple tasks and activities and need to switch between then quite often).

In today’s digital world lot of information is available but at times this can distract people.  Similarly as part of Agile best practices, where a lot of open conversation is encouraged for good reasons can cause and break sustained attention spans of people who want to focus long hours on writing their core logic. Both of the above can have a negative impact on the sustained attention span period of teams, makers in particular.

As per Harvard Business Review report, there are 22 interrupts that we get from others per day and as a manager this number is a daunting 50-60 interrupts per day, which means 1 interrupt per 8 minutes. And it takes us 23 minutes to get back to our 100% concentration.

So in order to improve team's productivity, one of the very essential and key aspects is to minimize interruptions. In order to do this we implemented something called FOCUS HOUR. Here are the guidelines of focus hour:

Focus hour was a time period in which team members were supposed to focus on their core activity of the day without getting distracted and without distracting others. We implemented this 2 times a day (1.5 hours each). Teams members were supposed to equipped themselves from all information before this period starts. They shall seek all guidance, help etc. before this hour. Other key guidelines for the Focus hours were:

1. Switch off email notifications
2. No Phone calls
3. Avoid breaks
4. No outsider to be allowed to come in the working areas
5. No conversations

We received a positive feedback about Focus Hours from all the teams though it is yet to be seen how exactly we can measure productivity improvement post this strategy has been implemented.

--------------X--------------