Web programming as a discipline is a relative newcomer, and has had a much different history than application development in other areas. The philosophy of Web developers has differed from that in corporate Information Technologies departments, and the technologies themselves are distinct enough that new tools and methodologies need to be developed. Additionally, Web developers would benefit by incorporating some of the processes used by existing developers, as the maturity of applications on the Web increases.
Web development has historically been the mistreated stepchild of corporate Information Technologies departments. IT departments were slow to adopt the Web as a communication and development platform, leaving the initial "proof of concept" developments to smaller groups in research labs and function-specific departments, and to students in academic environments. These departments typically did not have adequate resources to allocate to a proper development environment, so the developers were forced to make do with what they could cobble together from tools provided on the Internet. The initial development environments were initially predominantly Unix-based, using scripting languages such as Perl and Python, and perhaps some C or C++.
Eventually, some of the applications being developed were noticed by the IT departments. However, the integration of these applications into the Corporate IT departments was somewhat problematic. The philosophy behind the applications written by the early adopters did not match that of the IT departments. While the IT philosophy is one of integration, methodologies applied, requirements analyzed and priorities set, the "hackers" developing the early Web applications (which they probably did not even call "applications") espoused a philosophy of quick-and-dirty development, prototype-and-refine, implement-the-minimum, and forget about integration.
Additionally, the skill set needed to write Web applications was simply not present in the IT organizations. They were forced to "import" the skills by hiring people who had learned the technologies on their own, and who had been developing Web applications in the academic or end-user domains. Or, they simply embraced the emerging new development groups by outsourcing work to them.
In addition to the difficulties in integrating the human resources into the IT departments, there was a difficulty importing the technologies. The fast-and-loose style of programming and the use of Unix-based tools like Perl and dbm files did not integrate well with the corporate applications and database back ends and minicomputer or mainframe computing platforms.
With the advent of Microsoft's Internet Information Server, Active Server Pages, and SQL Server, as well as other sets of scripting language-database combinations, it became possible for even more workgroups to begin creating their own Web-based applications. Again, most of these were disconnected from the standard IT systems, both technologically and philosophically.
But the software industry had finally noticed that the Web existed, and now tools and technologies were being created quickly to fill the void. Tools such as Allaire's Cold Fusion, Lotus Domino, IBM's WebSphere, Vignette's StoryServer and Oracle's line of Internet-enabled database software, have all combined to give IT departments and developers a wide range of choices for software platforms upon which to build Web-based applications.
However, the maturity of the tools has simply not caught up with the tools used to develop standalone applications. CASE tools, analysis methodologies, development environments, software testing and usability testing applied to Web applications are all well behind classical application development.
Web development has split into two basic camps. The first still uses Unix and Perl, essentially working in a 3GL environment. The second uses one of the HTML-based markup languages such as PHP, ASP, or Cold Fusion, which can be compared with a 4GL environment. In fact the 3GL/4GL analogy seems quite apt. The Perl-based pages, while requiring more lines of code, require less of an investment in hardware to support. The HTML-scripting environments require more investment in software and hardware infrastructure, but allow small to medium-sized projects to be completed with fewer lines of "code" (admittedly hard to measure.)
However, large, complex projects tend to stretch the limits of what these HTML-scripting solutions can produce, and custom code must be written (in the form of server extensions, or other code written in lower-level programming languages), in order to support the more complex functions.
As the Web moved from serving HTML pages, to simple forms, to more complex and interesting Web-based applications, the development times remained very short. Users, developers (and their managers) began to expect applications to be developed on "Internet Time". Early on, this was possible because, in many cases, the requirements of the project were very simple, only one developer was involved, and any functionality was impressive to people who had not had such easy access to applications before.
However, as the nature of the applications being developed became increasingly more complex, the limitations of the CGI (Common Gateway Interface) model became more apparent.
In the CGI model, each request from a client (i.e. web browser) causes a separate execution of a program on the server, which requires the program to initialize itself, gather enough information from the incoming request data to reconstruct where in the application the user is, decide on the appropriate response, and deliver some kind of output to the user. This works nicely when the interaction is a simple "fill out a form" request, but it breaks down when the process contains multiple steps. Each step must encode enough information into the next request to allow the server to reconstruct the results of all the previous steps. This makes the use of standard development methodologies more difficult.
Even the use of new technologies such as servlets, fastcgi, mod_perl, and the like, while solving some of the performance problems, do not solve the problem of encoding and restoring the state of a transaction. The best solution to this problem thus far seems to be the use of cookies, or Java applets running on the client. Both of these have problems, in that some users distrust cookies, and even more distrust or dislike Java applets, or are unable to run them.
As Web applications become more accepted and more numerous, they are being used for functions that were typically reserved for closely controlled conventional applications. Some of these new requirements are: integration with existing corporate systems, security, robustness, use of real databases, and handling of financial information (such as Web banking and e-commerce).
However, despite these new "industrial strength" requirements, the infrastructure, expected timelines and development environments available to the developers are not able to keep up.
Tools available to developers now range from "Open Source" software such as Apache and Perl (and the many available Perl extensions) to heavy-duty commercial application environments such as Lotus Notes/Domino, IBM's Net.Commerce and Vignette's StoryServer.
Code re-use is one of the big problems facing Web developers right now. There is some re-use at lower levels (such as common Perl modules and utility libraries). However, at higher levels, code re-use is less common, since time is not being allocated to packaging functionality into useful high-level libraries.
Libraries that are developed rapidly become obsolete, and not enough time is spent updating the libraries, and the applications that depend on them. New applications either use updated libraries or entirely new ones, but the "legacy" applications (which may be only 6 months old) are still using the old libraries. Thus, old code proliferates.
Some code re-use is done through cut-and-paste. While this may solve the developer's immediate need, it simply creates yet another copy of the code, which must be maintained. Bug fixes made in one copy must be propagated to other copies, if anyone can remember where the copies are. In the case of Javascript, cut-and-paste reuse is almost completely unavoidable, since the Javascript code must be delivered to the client with each page, and designers are often loath to slow download times by putting Javascript into a separate shared file.
What developers really need, in order to be productive, and in order to meet the requirements expected of them are tools that:
Tom has a BSc in Mechanical Engineering from the University of Manitoba (1985), and an MASc in Mechanical Engineering from the University of Waterloo (1988).
From 1988 to 1996, Tom worked for Northern Telecom (now Nortel Networks) in various roles, initially as a software developer, in the Computer Integrated Manufacturing department at one of Nortel's Calgary manufacturing plants.
Then, in 1994 Tom moved to the newly opened Wireless Development Centre, where he administered a large network of Unix workstations, and supported the designers and researchers using the machines. In 1994, Tom installed Nortel's first web server in Calgary on Nortel's Intranet. Tom set up numerous Web-based "mini-applications" and Web-based interfaces to new and existing applications on other computers throughout the corporation.
In 1996, Tom left Nortel to pursue a career as an independent consultant, focusing on Internet technologies, and Web sites. Tom has worked closely with XOR Network Engineering, a large Web development and hosting company in Boulder, Colorado. XOR hosts many large complex Web sites, including all the Web sites for Times-Mirror Magazines (The Sporting News, Popular Science, Ski/Skiing, Golf, Yachting, etc.)
Tom has developed many Web-based applications, including on-line games, database-enabled applications, e-commerce sites (including an on-line auction), and Web-based applications which allow clients to publish pages on their Web sites without requiring any knowledge of HTML.
Tom has extensive experience programming in C, Perl, SQL, shell scripts, and Java, and has also taught Java programming. Tom recently completed training with IBM's Net.Commerce, in preparation for becoming a certified IBM business partner.
Last updated: Sat Feb 27 13:00:18 MST 1999
Tom Malaher
NetStart Consulting