OTUG Presentation Big Ball of Mud

OTUG's latest distinguished lecturer presentation was Joe Yoder and Brian Foote's Big Ball of Mud.

First, let me say that Foote and Yoder were the most entertaining speakers I've seen at an OTUG presentation. Their presentation style was to tag team the presentation. Each speaker would talk to a point or two and then the other would take over. Their approach really worked well. Foote and Yoder's energy levels seemed to stay at a higher level than it would have been with a single continuous speaker.

If there's one criticism that's resonated about the distinguished lecturer series recently it's that the presenters tend to be academics who do not speak to the same world that working software professionals live in. There's nothing wrong with speaking to the abstract, if not disconnected, world of academia. Sometimes I think that people come into these lectures with the expectation that they will receive a practical lecture that can be directly applied to their situations.

With that said, Yoder and Foote's presentation is clearly rooted in real world practical experience. If anything, Big Ball of Mud is the antithesis of the ivory tower lecture. It's all about the disconnect between what we say we are doing and what we are doing.

The premise of Big Ball of Mud is that software professionals aspire to produce software that is architecturally elegant, yet the product that we so often create is very different than those aspirations. As Yoder and Foote put it, "if [most] software were architecture, it would look like a shanty town." They liken software architecture to 'The Emperor's New Clothes' and that everyone is in denial that we are creating shanty towns.

Yoder and Foote call this ugly software mud. The dirty and chaotic messes of spaghetti. We all see mud in other people's code, and in that I've seen the monster and it is us way, we see it in our own code too. It's really a wonder that anything ever worked until we the noble craftsmen arrived, or is it.

They point out a few of the types of mud and the ways that it happens. Copy and paste, which they call metastasis. Big Bucket of Glue, or taking open source libraries from the wild and haphazardly gluing them into an ugly chimera of an application.

Yoder and Foote raised the question, "Why does stuff that sucks hang around so long?" It's easy to characterize the obvious negative traits of mud, but Yoder and Foote pointed out some of the things that aren't so obvious. The mud that we encounter works. If it didn't work, we'd ditch it. The mud that we see is sufficiently complex, otherwise we'd find the time to replace it with something beautiful.

Yoder and Foote showed examples of how a good enough solution now can be more valuable than a beautiful solution later. It kind of makes sense that for software to add value, it must be available.

Yoder and Foote addressed software craftsmanship. Brian Foote, talked about Amish furniture craftsmanship. He even wore an Amish hat for that portion of the presentation. If we think about Amish made furniture, it's beautiful and of excellent quality. We all would like furniture that is of that quality and is beautiful. We also buy stuff that's made of particle board from Ikea in droves. Sometimes the Ikea furniture is good enough.

The takeaway from the talk I got is that mud is the byproduct of a trade off from writing software in a way that costs less upfront. The mud is a technical debt that may, or may not, need to be paid down the road, but it is a debt that certainly does not get paid at the onset of its introduction. We can clean up mud through refactoring and replacing muddy software.

Foote and Yoder brought up a great point that automated tests will have your back when it comes time to clean up the mud.

We can strive to create mudless software, but it comes at a cost that becomes impractical. I'm reminded of a project that I was on. We set the goal to get our automated test code coverage at 85%, and we did. It was a worthwhile effort. Someone then thought we should strive to get the code coverage to 95% and we did at the cost of adding about 30% more time to our work. Was that 10% worth the effort? Is it worth 1/3 of a feature? In that case we let our idealism take precedence over the practical need to deliver something of value to our customers.

It seems that the image of the muddy boot architect may be an appropriate metaphor for software creators. Some mud is needed on the job site. We may have the opportunity to clean it up, but it isn't something that we need to avoid at the cost of delivering value.

As an observation, I don't think that the software that we are creating today, with the tools that we have available, is nearly as muddy as some of the code that was created in the past. We're getting better. As a community, I think that we are moving towards producing less muddy software. Just as I was thinking, nay fantasizing, about how we may reach a time where the mud we see is more like a fine layer of dust. Then Deion Steward had to bring up a question about concurrency and multicore development and I think that may be a whole other swamp.

EDIT: Please read Foote and Yoder's paper Big Ball of Mud.

Speaking: Integrated Test Driven Development

I will be presenting a use case for the Minnesota Scrum Experiences Group, a part of OTUG, on 2/23 at the University of St. Thomas. Murray Herrick room 155. Doors open at 6:00 PM.

My presentation will be a technical case study of using integration level tests to drive software development. My co-presenter, Ann Baumann Johnson, will give another perspective of the case study from the perspective of a project manager.

In our project we defined our technical requirements through executable JUnit tests at the application and integration level. We believe that by defining the requirements as such, the development team was able to deliver the exact program that was needed, even though the behavior of the program was never explicitly defined, we had no quality assurance team, and the consumers of the product were unable to test it until months later.

Although the circumstances surrounding this project were unique, the techniques we used are applicable to other software development projects.

My slides will be available on this site after the presentation.

More thoughts on the viability of evolutionary computation

I thought a little bit about the viability of using evolutionary computation in the field. That is, using Darwinian principles to design software.

In my writeup of Richard Gabriel's presentation to OTUG, I speculated that using evolutionary design may be a natural extension to the working specification or Test Driven Development principles. Essentially, if a specification can be provided in terms that can be fed into an evolutionary development environment, then the implementation can be left to evolution.

The downside to using evolutionary computation to implement our software is that not many humans may intimately understand how the implementation works. I began to think about how different letting a computer implement our designs is versus what many companies are doing with offshore development.

The model of offshore software development with many companies is to keep the architects and high level designing engineers stateside and send the work in the trenches overseas, i.e., all the finer grained detail work. I believe that many organizations that adopt this type of model are naive in thinking that it will simply work without a significant investment in establishing communication protocols between the high level designers and the people responsible for actually implementing the software.



The designers become an intermediary between the business/business analysts and the implementing engineers. The high level engineers take the requirements from the customers and give them to the engineers. They rely on their ability to communicate those requirements. Their people skills if you will.

The pitfall that friends of mine have encountered when trying to operate as the designer/intermediary reads like a Threes Company episode, i.e., breakdowns in communication cause considerable problems that seem to get discovered so late in the product development cycle that they require either a heroic effort to keep the project on schedule or they cause delays in the product's release.

People complain about the code coming from India being sloppy and difficult to read. It is clear that little attention is paid to considerations, such as future maintenance. Much of this I believe is due to cultural differences between Americans and the people who are hired to write software.

One must question the reasons why an offshore model of software development is implemented. Some organizations do this because sufficient qualified domestic help is unavailable. Company's are trying to save money by shipping work to a cheaper place. That's the theory, but the practice is wrought with more hidden costs and difficulties that often make offshoring more expensive than just doing the work in-house.

If the motivation is having cheaper workers regardless of the added burden and workload of the senior people in house, one should consider the difference between the added workload of the architects and high level designers and specifying the program parameters to a computer to design the software.

If the difference between specifying the acceptance criteria to a program is not sufficiently more expensive than specifying the acceptance criteria to a team of developers on the other side of the globe.

Given sufficient specification and evolutionary cycles, the resulting software will outperform the software that is developed by any human designer.

I think the next step in evolutionary computation is a system whereby human designers specify an interface and behavior and fitness model for software components, and the computer figures out how to meet those specifications as closely to the specification as possible.

Letting a computer handle class implementations, or even class level optimizations, seems like a natural progression from the current state of software development. Consider the level of abstraction that object oriented development and programming in domain specific languages has taken us. We're already trusting much of the implementation to the computers. It only makes sense to push the bar and see what we can do with another level.

Richard Gabriel's lecture at OTUG: 2/9/2009-- my mind is blown

Gripes first, praise second.

 
First gripe: it was a bummer that Jared Richardson was in town to speak at the Twin Cities Java User Group on the same night. I would have wanted to see him speak. EDIT: Jared was kind enough to record the presentation, link here.

Second Gripe: I hate slow reveals. They waste everyone's time. Gabriel's lecture was a slow reveal. The first half of the lecture built context into the concept of Ultra Largescale Systems(ULS), and descriptions of Metadesigns for building them.

I'll cut to the chase of the first two Metadesigns: 1 is waterfall or what I heard Neil Ford describe as a Big Design Up Front, BDUF. Metadesign 2: is stepwise design, or what we're calling iterative development, or Agile development.

I think that it would have been sufficient to state that Metadesigns 1 and 2 are insufficient for developing ULS and proceed straight to the meat of the lecture. Effectively, I think that over half of the lecture was spent recanting things that we already know.

That is the conclusion of my negative criticism.

Conceptually an ULS is a system that is beyond a human's comprehension. Some examples that were given are the entirety of New York City, a healthcare system for every living being on the planet, the entire ecosystem of the planet's oceans. An Ultra Largescale System is by definition beyond the design and understanding abilities of humans.

How do humans build such systems? Metadesign 3: evolutionary design, or using evolution to design the solution.

In my opinion, this is where the lecture started getting interesting.

Never tell people how to do things. Tell them what to do, and they will surprise you with their ingenuity.

I must confess that Genetic Algorithms are a curiosity of mine. I like the idea of tellin a computer what a want, giving it some information and letting the computer come up with a solution.

The example Gabriel gave is Adrian Thompson's Exploring Beyond the Scope of Human Design: Automatic generation of FPGA configurations through artificial evolution.

Thompson's experiment was to use the principles of evolution to dictate computer logic. In the case of Thompson's experiment, the desired trait of his circuit is to produce a circuit that can differentiate the following tones: 1 khz, 10 khz, and every other tone. The building blocks of the circuit are 100 FPGAs, or a 10 x 10 matrix of relatively simple programmable logical components.

A fitness algorithm was set in place that mimics the principles of evolution. Circuit designs that are the most fit carry their traits on to the next generation. Ones that are not deemed to be fit, do not carry their genes on.

...how noble in reason! how infinite in faculty! in form and moving how express and admirable!in action how like an angel, in apprehension how like a god!
--Hamlet, Act 2, Scene 2

The evolution was allowed to run for 5000 generations. The first thousand or so produced circuits that performed poorly or outright failed. The circuits improved over time though. The circuits evolved to outperform human designed circuits.



The circuit was analyzed after the experiment and some very curious details emerged. Only 21 of the FPGA gates were used. 79 of the 100 gates could be removed completely from the circuit without affecting the circuit's performance.

5(in gray) Gates appeared to serve no logical function. However, if they are removed from the array, the circuit fails to work. The same circuit's performance degraded when it was duplicated on another array. On software it also was degraded.

The circuit adopted feedback loops and other characteristics of an analog circuit.

What's truly amazing to me is the design. It is something that is outside of the realm of human comprehension. Nobody would think to design a circuit this way because it doesn't make sense to include seemingly disconnected and worthless elements.

I shouldn't speak for others, but I would never think to include the extra elements of the circuit because it would strike me as wasteful and cluttered. The result looks like a maintenance nightmare.

The circuit is truly amazing. Experts don't have a clear understanding of why all the elements are necessary. Only by removing them are they able to see that they are.

My own speculation is that there are small variations in the FPGA gates and only through evolutionary exhaustion were they found.

This is fascinating to me for many reasons and the implications of using the principles of evolution through natural selection to design solutions to find optimized solutions to problems.

Consider the current results of evolution through natural selection. Life, is truly amazing. Consider the diversity of animals, plants, and microbial life. Life is able to adapt exceptionally to myriad environmental conditions. Life at an evolutionary level is unforgiving, you fail, you die, and there aren't any little ones like you after you're gone.

The same rules work surprisingly well with software. I think that it can be complementary to Test Driven Development. I wonder if evolutionary design could actually replace the human part of development.

Consider the differences between TDD and evolutionary software development: both can state their acceptance criteria upfront and both provide a solution to the problem until the acceptance criteria can be met.

The advantage that evolutionary design has over human designed solutions is the absence of any human prejudice. Evolution is really a results oriented environment--maintainability, comprehension, beauty, personality, and all the other human influences are not part of the equation--well unless they are defined as part of the acceptance criteria. Given enough generations, an evolutionary design will produce the most fit solution as defined by the acceptance criteria.

By letting evolution manage the implementation of our goals, implementations that exceed the abilities of human comprehension can be produced. In essence, the programmer is god like in her ability to create programs that exceed her own ability to create a program.

Gabriel said that the results of evolutionary design is troubling to creationists and evolutionists. The trouble to creationists should be obvious. It shouldn't be a concern to the rest of us though. There are plenty of examples where observable reality should be troubling to those who buy in to a religious dogma. For example, if the universe is only 6,000 years old and we are able to observe stars that are 5 million light years away--and much further for that matter--and we know the speed of light to be constant; how can one honestly continue to believe that the universe is 6,000 years old. Something has to give.

For those who believe in evolution, Gabriel asserts, the results of evolutionary design should be troubling because we, humans, may not be able to explain the results. Are we to believe that an intelligent designer stepped in and created the results of the evolutionary design? The results actually affirm my belief in the viability of evolution.

But I digress, evolutionary design certainly does raise some issues that some may find in conflict with their faith. Gabriel did mention a computer scientist who refused to work with evolutionary design because, presumably, of her beliefs.

Other questions arise though. Who should take ownership and responsibility of the results of evolutionary design? Certainly the first person to claim ownership of something that can be monetized will want to claim ownership, but what about responsibility. Will that person accept the liability of the software that they claim to own. What if an evolutionary designed program turns out to cause people harm? Who should be held liable? The concept of ownership can be challenged by this method of design.

Second Life for Software

Gabriel concluded with a proposition that for evolutionary designed software to take the next step and be integrated into an ULS an environment where software can interact must be created. He suggested a Second Life be created for software. His proposition is a virtual world where humans and software can interact. In this world software would take on physical characteristics of volume, mass, odor, color, shape, and sound.

In this world software would be free to evolve. The software that is useful or beautiful to humans would thrive. Gabriel kind of lost me at this point. I'm going to have to think about it for a while.

8/19/08 Otug Meeting: Agile 08 Retrospective -- My Retrospective

The Object User Group hosted a retrospective of the Agile 08 conference. A very good group of approximately 25 people attended. Among the attendees were Tom and Mary Poppendieck.

What a pleasant surprise to finally meet them. I had a chance to chat with Mary. I didn't mention that I've been publicly admiring her writing on this blog. NOTE: Mary, you write excellent essays.

Agile 08 sounded like a wonderful conference. The vision for the conference was that of a large multi-stage music festival. Each stage hosted a theme, e.g., the main stage had the more well known speakers, but there were also stages with themes of leadership, coding, quality, etc. There was also live music that was provided by the attendees. It sounds like a wonderful time. Agile 09, is going to be in my home city of Chicago. I may just need to attend. Hopefully I will think of something valuable to say and have the opportunity to present.

One of the themes that surprised me is the movement away from iterations. Instead of having iterations, some teams just have regular releases based around a minimum useful feature set.

The advantage of having a system like this is the release is scheduled around the time to develop the most valuable features. If it takes 3 hours to develop/test/approve a feature that adds sufficient value to the business to warrant a release, why not have a release based on that feature instead of waiting until the next iteration? Iterationless development was said to be well suited to supporting products that are mature or in maintenance mode.

Iterationless agile development was recommended for mature teams that enjoy good relationships with their stakeholders. As Tom Poppendieck said, "Iterations are because people don't trust [each other]." Lacking iterations requires trust among all parties involved.

Another theme that was discussed is the practice of promiscuous pair programming. PPP, or 3P as I just called it, is a practice of pair programming and regularly changing pairs within a day. More experienced programmers would rotate between less experienced programmers. One of the benefits of doing this is gaining the benefit of having new perspectives from the less experienced programmer and knowledge transfer from the more experienced programmer.

The interval that was said to be optimal was between 45 minutes and 90 minutes. The discussion digressed around David Hussman's pomodoro, or tomato timer.

Hussman employs the use of a tomato shaped cooking timer to set work/break intervals. Using regular work/break intervals is a way that he believes allows for people to sustainably work well throughout the day.

I have never personally worked in a strictly paired environment. When I was interviewing at an XP shop, I asked some friends for feedback on the practice and most of them said that pairing for them became a pain point. They said that they would get sick of working so closely with the same person. I think that promiscuous pairing would do a good job to mitigate that risk.

The topic of distributed development was another well covered theme. Mary Poppendieck made a statement that strengthens my view on offshoring. She said that European companies are offshoring because there is a real shortage of qualified workers. They want talent and are looking to add value to their company.

One company, which I don't believe was named structure their organizations to support better integration between their European and Indian sites. One of the things that a company did are to create identical facilities. Another thing was having the Indian team start working at the European facility for weeks and then transfer to the Indian facility. They also have an ambassadorship program. The program gives developers the opportunity to work at the other site for a period(a month?). The results of the program are teams that do not suffer many of the challenges that many of us do.

This is a stark contrast to my own experience working in situations where the company is offshoring on the basis of cost. If the motivation is to reduce cost first and provide value second, it is the beginning of a race to the bottom.

How could we have a discussion about agile development with Mary Poppendieck without discussing Toyota? We didn't. Mary praised Kenji Hiranabe's presentation on new car development at Toyota. Some of the points that she noted about his presentation were some leadership traits and how they are perceived, positively/negatively, by Toyota's culture. One surprising negative trait is charisma.

Another interesting facet of Toyota's culture is that in meetings it is expected that the bad news be discussed first. This is a big contrast to other corporate meeting cultures, e.g., Target Corp, where meetings begin with a round of complimenting each other.

Mary described Toyota's product creation practice as a three part process: product conception, product development, and product production. Each stage must be completed before the next may begin. In each stage of the process a different schedule/finish criteria is used.

For Product Conception: it will not finish until it is complete. Product definition is so important that development will not begin until the definition is finished. I see this as being akin to creating a description of what characteristics the product will have.

For Product Development: it will be complete when it is time. Development is schedule driven, the finished state of the developed product will be finalized when time is up. This is akin to developing an early workable product and enhancing it with as many elements from a prioritized feature set as the schedule will allow.

After the meeting, about twenty of us enjoyed fantastic discussions and free iPhone app demos at the Davanni's pizza.

05/20/2008 Ted Neward's presentation: The Busy Developer's Guide to Scala

Ted Neward gave a great presentation on Scala last night at the Object Technology User Group. If you have the chance to hear him speak I recommend it. He's a cool guy.
I wish I had prepared better that evening so I could join the group for pizza after the presentation, but that is my only regret.
The following is what I learned from the presentation and beyond that are a few of my own thoughts on the value and future of Scala.
Scala is a functional language built on the Java Virtual Machine. Like Groovy, Scala can use existing Java objects easily, and Java can use Scala functions with a little bit of work.
What's so cool about functional languages and Scala? That's really the big question. The strength of functional programming is that every operation in a functional program can be thought of as a mathematical function. You can think of it as an implementation of Lambda Calculus, at least that's how I'm going to think about it until I find it doesn't work.

For example: The operation 1+2+3+5 be thought of as
theValueOne plusTwo plusThree plusFive
which gets evaluated to
theValueThree plusThree plusFive
which gets evaluated to
theValueSix plusFive
which gets evaluated to
theValueEleven

In functional programming functions enjoy the same status as objects. Or another way to think about functional programming is, everything is a function, values can really be thought of as functions that returns a value.
What's the advantage of functional programming and what are the advantages of using Scala? One simple advantage of Scala is it's a new language that was conceived within the context of how programming languages are used today. What do I mean by that? Marten Odersky, the principal designer of Scala took careful index and consideration of the idiosyncrasies of the Java language and improved upon them. There are many restrictions and constraints in the Java language that we just accept because we've grown accustomed to them.
Java's a maturing language. Like people, maturity isn't all win. Like with people, programming languages pick up some undesirable traits over time. Older, wiser, and more capable, yes--but Java's starting to develop a gut, it's butt is sagging, and some of the new features are like the type of outpatient cosmetic operations that seem to be so popular with the over forty crowd.
Languages created on the Java platform are more like genetic engineering than cosmetic surgery. Designers can take the pieces of Java DNA and work them into their new language, they also have the opportunity to leave out the strands of Java DNA that they don't want. Only time will tell whether a bunch of Frankensteinien monsters come from this experiment. I'm inclined to believe that the LDL reducing ribeye steak is around the corner.
With new languages we have a great set of opportunities. First, we can see how languages are being used and develop features to accommodate those usages, e.g., XML is a first class citizen in the land of Scala.
We also can see areas where a programming language can be improved. For example, Scala has implicit typing. Consider String s = "I'm a String" verses s = "I'm a String". Is there any doubt as to what the type of s is? The Java language seems to think so, that's why we need to explicitly declare the type of all objects.
There is enough Syntactic Sugar in Scala to rot your teeth out.
That's all well and good, Groovy does the same thing. Correct. What does a functional language provide that other languages don't? Scalability. Consider this: a functional language composes functions by passing functions as arguments.
This is like a certain hot topic debate where a group of people rhetorically denounce a theory because it is not a fact. Ken Miller has an eloquent explanation for why a theory is more useful than a fact. To cut that 2 hour lecture to the chase I'll paraphrase: "A theory is more powerful than a fact because a theory can be utilized to derive facts."
A function in a functional language is like a theory. We pass theories around and then apply them to values to derive facts. For example: we may employ the theory of gravity to derive the time that it would take an object to drop 100 feet or 10 feet. If you think about it, most theories are compositions of other theories. Data points are applied to those theories to derive results.
Functional programming is no different. We define functions that are composed with other functions in a process called function currying.
The application of a value to the curried function is independent to currying the function. That is to say, we can curry functions without transforming any data.
As such, thread safety and concurrency are not issues in the functional programming world. That is where I see Scala making the biggest impact. When the Java platform can be used to perform distributed and concurrent computations then we will see a fusion of Java's ubiquity and the power of parallel computing on a scale that the Java community has never seen.