This document describes the most important (but not all) aspects of project methodologies from the logistical point of view. The following topics are discussed: the specific nature of a software project that make it different from hardware, the importance of detecting "weak links" and the human issues, the leadership issues, the interfacing between hardware and software sub-projects, and the development process issues. The subject not covered here concerns the coding techniques (see separate document "Software Coding Guidelines for C").

The main difference is that software development is much more serial than the more parallel nature of hardware design.

The second difference is that the quality of the code is much more dependent on human skills and experience, than in the case of hardware projects. It is possible to design a decent circuit following component data sheets and textbooks; unfortunately it is not so in software. Nowhere (except in entertainment) is the role of the "star performer" so important as in software. Quality software is so often a result of a work of a single top programmer, either directly or by managing, that a company may experiance some real difficulties should such a person quit. There has always been attempts at changing this situation by implementation of various quality verification procedures, team work arrangements, formalized process structures etc, none of wich was successfull. Even very big software companies such as Microsoft rely heavily on "stars" (and reward them handsomely!).

Software development consists of a series of tasks that typically, are much more interlinked than the elements of hardware development. For example, in hardware projects, there is much more scope for substitution of components and modules. It is also much easier to parallel certain tasks through sub-contracting of parts and services, and to spread the work load among several team members. Hardware tasks are also easier to objectively verify on completion, since the performance of a sub-circuit can be measured by instruments.

In software projects, paralleling is more difficult even within a team; it is difficult to verify the performance of a subcomponent, and in some cases almost impossible. Quite often, the proper verification can only take place late in the process, during final stages of system integration (e.g.: NASA Mars Polar Lander case)

Out-contracting of some parts of the projects is difficult and also prone to failures; even when it does works, it still might not really off-load the project resources, since the effort of integrating & verifying an out-sourced software component can be comparable to its development. Such a situation puts a lot of pressure on the project manager.

Since the software development is so difficult to meter, it is extremely important to select members of the team properly of the similar level of expertise, since a weak link (e.g.: an inexperienced programmer) might be difficult to identify in the course of the coding. Additionally, a performance measurement based on the amount of generated code is bound to produce meaningless or worse - contradictory results. I have seen projects, where most of the code was written by the least experienced programmer. Not only such a situation is wastefull, since more experienced team members then end up spending inordinate amount of time later on, debugging someone else's code, but also it may lead to personal hostilities and conflicts. It is important to reserve one's judgment till the very end, and to refrain from rewarding any team member for prolific or quick coding. On the other extreme, punishing a suspected team member during the course of the project is not a good idea either; it destroys the good team work and it creates an atmosphere of mistrust. It is also very easy to misjudge and hurt somebody undeservedly.

In terms of success rate and quality, both hardware and software project rely heavily on the individual creativity of designer. However, because of the lack of adequate performance-measuring methods in software design, the existence of "week links" is a MUCH bigger issue in that area.

The best, tried and recommended method of handling a "weak link" situation is to impose a strict boundary on the amount of code that a team member is allowed to write. Anybody who has finished his/her code ahead of the others, should not be allowed to code more, but instead should spend the rest of the time writting debugging routines and debugging his/her own module beyond the usual verification requirements.

Incidentally, the best place for junior programmers is definitely not as team members, where they might become the "weak links", but in a much safer position as trainee-apprentices, or as auxiliary support programmers assigned to individual members of the team.

One should also bear in mind that all people exhibit some weakness in certain areas. For instance, a programmer who might not be the best at designing & debugging could be really excellent and well worth its money in writing manuals or in testing.

a) The leader has not got much executive power but at the same time has all the responsibility of a team/project leader, such as getting the project properly organized & planned and to cultivate communication with the upper management,

b) The leader has both power and responsibility.

The problem is, that in case (a) the position of the leader is really redundant under the engineering or project manager, who has the real power to run the show. Even such team/project leader specific tasks as getting the project properly organized & planned and to cultivate communication with the upper management, can be off-loaded onto auxuliary adminitrative personnel (e.g. planning) or easily done by the manager (e.g. team communication with upper management).

The problem with (b) is that it makes the role of engineering or project manager unclear or redundant; which may often promote conficts.

The best recommendation is to flatten the hierarchical structure of project development by refraining from appointing a team/project leader. Instead, it is strongly recommended that a software project is lead run by a real manager.

Since the software project, especially in their early stages are notoriously difficult to meter, it is highly advantageous that the manager has got sufficient technical (software) project-relevant experience, and posseses enough understanding to be able to assert the situation, without over-reliance on internal or external reporting.

Note that reporting by team members tend to be unreliable and biased; people don't normally like reporting bad things on each other, even if they should; and unfortunately, some people do like it, even if they shouldn't. I have seen a company with a peer-review process firmly established, where the staff was obliged to file performance reports on each other regularly. The result was a gradual erosion of trust, over the period of years, and a high level of frustration leading eventually to the resignations of the most experienced from among engineers and managers.

It has been the experience of many high-tech businesses that some among the successful and experienced engineers can quickly acquire middle management skills through formal training and through experience, given a chance. (Note that the loss of a hands-on designer to management can be remedied by a new hire).

On the other hand, the opposite is seldom true, that is, the appointment of a professional middle-manager with no specific technical expertise in the current project, does not normally lead to quick acquisition of the necessary technical skills which ultimately may lead to all sort of problems.

It is very difficult to acquire enough technical expertise (or any other expertise, for that matter) and it takes a long time, in order to be able to run research and development projects efficiently, with sufficient confidence and respect from the staff. The chances are that a professional but non-technical manager in a high-tech R&D environment is likely to face hostilities and dis-respect from his/her more experienced and more knowlegable subordinates, who might regard him/her as a hurdle rather than an asset.

Note that above comments do not deal with selection criteria for top level management, where entirely different kind of skills and methods are required, where leadership qualities, intuition and financial expertise is a lot more important.

In a case where no middle level manager with suitable expertise is available to run the software project, and an internal promotion is not possible, it is thus strongly recommended to place the entire project with an external contractor.

To summarize this section:

1. Advantages of internal promotion:

- Has already got enough technical expertise and formal education.

- A formal middle level managerial training can be done relatively quickly and inexpensively.

- Salary increase may be relatively modest (in comparison with a hired middle level manager with a proven track record)

- It is likely to result in the continuation of existing company culture and good practice (if desirable)

 

2. Disadvantages of internal promotion:

• May cause some begrudging within the staff, at least initially.

1. Advantages of hiring a middle-lever manager:

 - It is more likely to result in the dis-continuation of existing company culture and bad habits (if change is required), and bringing in some fresh ideas from the outside.

- It may initially be more pallatable for the team (i.e. no one was promoted "above" the others)

2. Disadvantages of hiring a manager:

Since the software development is such a highly creative activity, by its nature requiring people who are highly skilled and self-confident, it is very rarely that the Ordering and Directing style would work or would be required. A Guiding and Coaching style of leadership is what normally works. It is not a good idea to have team members who require ordering or directing , if such people need to be involved, it is recommended that they be organized as external resources to the team, or assigned as auxiliary help reporting to individual members of the team (i.e. aprenticeship idea, see previous section).

Additional to the usual duties, one of the most important jobs for the software project manager is to have the project structured in such a way as to maximize the opportunity for monitoring the quality and the performance of team members at certain milestones, especially in the initial stage. The purpose is NOT to create a competitive atmosphere, but to discretely watch for a "weak link" (if present). One method of achieving this is to divide software into multiple parts (i.e. low-level hardware I/O, middle level numerical signal processing, high level user interface & graphics etc.). Then each software part would have a specific (and independed from other modules) debugging & verification stages, utilizing some kind of simulated environment. There is a certain overhead associated with this solution, i.e. the simulation software has to be written in addition, but it is well worth the effort.

Among some most common solution, we shall discuss the following four:

• Easier to select suitable managers.
• Individual groups are easier to manage.

• More difficult to coordinate the overall project. May require appointing a higher level superviosor (e.g. R&D director etc)
• A danger of software and hardware groups becoming too separated, which may lead to unnessessary competition, conflicts and blame-shifting.

• Only one project manager is needed.
• Better synchronized development, more transparent & easier to plan.
• Easier to shift or to allocate resources.

• More difficult to find a middle level manager proficient and experienced in both hardware & software.
• Manager has more responsibility, and more stressful role. Requires special skills to make software and hardware engineers cooperate.

This option is really viable only if the hardware part of the project is relatively minor, or if it involves outsourcing.

• Only one project manager is required
• Less distraction and less competing for resources.

• Hardware issues may get overlooked or forgotten.
• More difficult to resolve hardware problems, if such occur.

This is a viable option only if the software part of the project is relatively minor, or if it involves outsourcing.

• Only one project manager is required
• Less distraction and less competing for resources.

• Software issues may get overlooked or forgotten.
• More difficult to resolve software problems, if such occur.

Before the project begins there must be some initial stimulus (a new product idea, a request to upgrade etc.), and there must exist a direct or an implicit (quiet) approval from the top management. The distinct phases of a hardware project are:

a) Feasibility study & research (technical+marketing)

b) Drafting design & specification

c) Lab prototype implementation & testing

(Note that the software used at this stage may but need not yet be the same as the software being developped at stage 'C', e.g. a normal practice is to use higher level packages such as LabView, Mathlab, etc)

d) Final prototype implementation, integration & testing.

Note that the system must use the production software release (may be "beta"), such as the one being developped at phase 'D'.

e) Production implementation.

This is the specifically hardware phase which has no close equivalent in software. Production implementation is a complex phase, which is othen split into mechanical and electrical. The software team should use this time in parallel for the completion of phase 'E'.

f) Test, field tests & launch

This phase needs an integrated hardware+software system, with the hardware taken out of a pilot production run, and a non-beta software release. Any problems found at this stage should cause the recycling back to 'd','e', or 'E'.

Note: verification is not listed as a separate phase, since it is a continous process necessary at every stage.

Software part of the projects goes through some very similar phases, usually but not always in-synch with the hardware phases:

A) Feasibility study & researching (i.e. selecting the best platform, operating system, compiler, libraries etc.)

B) Drafting design & specification of the overall system ("topdown") AND of some critical modules ("bottom-up").

C) Coding and debugging a prototype version of software.

(Note that the hardware platform used at this stage may but need not yet be the same as the hardware platform being developped by the hardware team at stage 'c', e.g. a normal practice is to use evaluation boards)

D) Re-writing a prototype version into production release. (Note that in some cases this means writing new code from scratch! At this stage, the software should not use an emulator but rather the intended hardware, such as the one developped in phase 'c' or 'd' ).

E) Debugging & verification of the production release. This should be done exclusively using the final or close-to-the-final release of the hardware, such as one developped in phase 'd'. Note also that the last phase which would have been "F) Test, field tests & launch" is common with the hardware phase 'f', therefore it is not listed separately.

The overall project is structured by interleaving the software and hardware-specific phases, such as a,A,b,B,c,C, D,d, e,E and the phase 'f' which is common to both.

Note that the stages A,B,C don't need to be fully synchronized with a,b,c. On the other hand, phase D must follow c and precede d. See Fig.1 below.

The above-presented diagram has been done only for the purpose of illustrating the distinction between the software and hardware development flow. It is not the intention of the author neither to suggest nor to advocate any particular formalized structure for the development process. In fact the opposite is true, one can make a strong argument for not formalizing it.

Dividing project into phases or stages is useful tool for metering the progress of the project development, and for defining certain easily identifiable milestones for budgetary and control purposes. However, one should caution against to detailed a planning. The author has seen a company that employed a very detailed and formalized system of R&D project planning, with mandatory 50-page reports and approval by a committe at the end of each phase, which resulted in so much administrative overhead that the average project time was nearly doubled, and the overall number of new product introductions withered to a fraction of the past. It is not possible to plan for all the unknowns, nor it is possible to eliminate all risk through planning (though it IS possible to cover-up the risk!). There are two separate issues here: over-planning and over-formalizing the R&D process. Let us clarify them further.

The critical thing is to get the right amount of it. The danger of too little planning is that the success of the project depends solely on the vision, judgement and the initiative of the project leader or the project manager. Though such a system often works well in software, should there be any problem (technical or otherwise) there would be very little the higher management could do to rectify it. Proper planning is also a practical necessity for big projects, especially involving hardware, in order to be able to allocate, to budget and to coordinate resources.

Dangers of overplanning are:

• Increased administrative burden imposed on project team members, especially on the project manager.
• Increased focus on timeline and budget, often disproportionately, at the expense of technology, product reliability and marketing.
• Increased fixation on the current product spec, which is often disadvantageous if the market or technical requirements change in the course of the project. In the high technology business, particularily in software, flexibility is extremely important.
• Overplanning encourages robotic behaviour, which often frustrates the more creative members of the team. Fostering creativity is one of the critical factors in successful software (& in general) development.
• Allows some team members to avoid initiative (e.g. "can't do it, it's not in the plan!")

Dangers of over-formalizing the software development process:

• Increased administrative burden imposed on project team members, especially on the project manager, and on the supporting staff.
• Reduction of executive power of the project manager.
• Increased, often disproportionate focus on the fulfilment of bureacratic requirements at the expense of technology, reliability and marketing.
• Leads towards rewarding a formalized behaviour and conformance to procedures, resulting in alienation of the more creative members of the team.
• Allows some team members to avoid initiative or slow down the work using administrative tasks to fill up the time.
• Allows groups or divisions external to the team to avoid or slow down the support work using bureacratic or procedural excuses.

Note: It is hard to give definite receipes and general recommendations since the specifics and needs of high-tech projects and businesses can vary dramatically from case to case. The conclusions presented below may, but need not be applicable in a particular case.