A Life of Software Development - Curse of the Data Structures

1986-1987

Once it was clear that we would have to develop financial software if we wanted to get into the market, the way ahead was more or less determined. I had got accounting lessons as part of my education and had a quite good grasp of accounting principles.

At that time, most companies developing accounting software used really archaic development environments left over from mainframes or minicomputers. They also lacked a lot on the user interface, since most of their interfaces were line-based and could handle one line transaction at a time. Having been into complex data structures and using good principles in structured programming, I was interested in utilizing these in real-life programs. Using Pascal - a modern programming language at those days - had its advantages.

One challenge we had at the time was to be able to represent the account codes in a proper, hierarchical data structure. Because the way accounting systems were set up, we had to have a tree-based structure, and obviously not a binary tree. Binary trees were easy to handle, but inserting new elements and re-balancing the tree took a lot of time.

Each main account would have an unlimited number of sub-accounts and we had to access these really quickly in memory to prepare ledgers and similar accounting documents in a relatively short time. At that time we were not thinking of using ready-made databases, first because they were quite expensive and we could not afford the license fees, and also because most of them did not perform very well anyway. These limitations quickly brought us to the use of n-ary trees in memory. While searching for a good way to represent these, we were introduced to the concept of B-trees. These were superstructures arranged in tree format where the upper leaves would correspond to index pages which would store a link to another page in the tree that stored keys in a certain range. In a sense the top-level pages corresponded to the "Table of Contents" in a book which gave us fast access to the lower-level pages. At the lowest level, the actual keys and the corresponding data structures were stored. One problem with this structure was that it had to be re-balanced from time to time, which could take an unknown amount of time to process.

Since we did not have access to a database, let alone a relational database with SQL support, we had to work with flat files. In addition, we used linked lists in memory, through which we handled the relations by keeping record pointers between consecutive record belonging to the same list and saving each record in a sequential-access file. The "curse" of this data structure was that if something happened while saving the data into these flat files (such as electricity cutoffs, which happened regularly those days) then the data would be invariably corrupted, since links would point to invalid entries. This could happen when there were programming errors as well.

We released the Accounting Module and suddenly encountered a constantly increasing demand. The relatively easy user interface that Turbo Pascal enabled and the fast operation due to the use of clever data structures in memory differentiated the product from its competitors. However, the flat file structure was causing frequent corruption of data structures and we had to bring in files of the customers and manually fix them. In time we would develop automatic tools which could make this task of fixing corrupted files a bit easier.

While transforming ourselves to a provider of accounting systems, we felt the need to grow. Up to that time, all the programming was done by ourselves. We got a secretary, then a support person, then an office boy, and right afterwards two young programmers.

In the meantime a major computer seller had approached us, asking us to develop a special version of the Accounting Module for them exclusively. They also asked us to add a couple of other modules, namely Inventory Control, Invoicing, Order Processing, etc. They would provide the financial expertise, assigning one of their experts to us for requirements capture, but they wanted this product to be bundled with their computers. It was an attractive proposal for us, a fledling young company, and we signed a contract without thinking too much about future implications. Since we had some deadlines with monetary effects, we started working hard on this special program. We would find out later that the contract implied quite restrictive terms for all of our software development. We produced the software and started testing, supported by the financial experts of the ordering company. As we were supporting their own developers who would eventually take over the software, we were also building a new software development group for our future products. We built similar modules with information from internal sources or other experts, and prepared to market the new Integrated Suite of Financial Applications.

We were very surprised when we got a formal warning through a Notary, asking us to stop all software development in the area of financial systems. The warning was drafted by a very well known Corporate Lawyer whose name frequently appeared in the press. When we contacted them , they said that they just wanted to protect their investment. Although we agreed that we had signed a contract to produce them the software, we did not intend to stop all software development in this area, since this was the most sought for area of software. Through a friend we got ourselves a good corporate lawyer, who contributed to public cases a lot. He would take on the task of opposing big corporations when they tried to bully smaller rivals or individuals, so he was a man of principle. He read the contract and told us that, although the contract could be interpreted as covering all software development, he thought this would be a very restrictive contract and thus would not normally be allowed under normal contract law. It was clear that we had been misled into signing this contract, because of our lack of understanding of contract law. He sent a counter-warning and the battle was on. After many months of warnings and counter-warnings, they figured out that even if they could have an injunction for us to stop all software development in this area, it would be very difficult to enforce, furthermore they needed us to support their exclusive software effort. In the end we revised the contract to provide more support and they accepted the removal of the restrictive clauses.

It was now possible for us to increase the scope of our activities, especially since the contract resulted in a much-needed cash influx to the company. Building on the success of our Accounting Module, we proceeded to develop the Integrated Financial System, with Inventory Control, Order Processing, Invoicing, Checks and Notes Management and eventually Materials Management (a precursor of today's Enterprise Resource Planning systems). We would now establish an official Software Development Department (with me as Director) and populated it with several young software developers. We also had a production department which constantly produced the packages (one by one writing the floppies and packing them up, floppies were 5 1/4 " and 3 1/2 ")

One challenge was to convince major PC sellers to bundle these software with their PCs. It was very well known that companies saw software as a part of the computer and did not want to spend extra money on software. It would be an advantage to include the software in the price of the computer (with some discount) and show it as part of the complete system. However, most of these PC importers had agreements with other developers. It took us several years to convince them - one by one - that we had a higher quality product, better GUI, better customer service and eventually they started selling our software, sometimes exclusively, sometimes as an alternative to other companies' financial software.

Another challenge was to provide training and good user documentation. This would eventually grow into an industry, with companies specializing in providing training in these tools, but in the beginning we had to do all these ourselves. The company was growing at an uncontrollable pace. The company office now held two full stories in a building, whereas we had started with a single room, maybe 5x4 meter square.

Once the sales of the Integrated Package started growing, we could also go back to one of our earlier goals : We produced a Macintosh version of the Accounting Package, albeit selling very few of it.

But working once more on the Macintosh would have an unexpected effect on my software development skills: I was introduced to a totally new concept in programming, namely that of Object-Oriented Programming.