ProjectPipe bug fix update

June 29th, 2007 by csimms

The server was upgraded this morning with the following bug fixes:

  • If you’re viewing a ProjectPipe feed in your RSS aggregator, and you launched an item in your browser, then it sent you to a bad link.
  • Keyboard shortcuts were too aggressive and control-c (the standard copy/paste operation) was instead treated as a “create child” inside ProjectPipe.
  • The hoist icon in MS IE looked ugly.
  • Round-trip export then import of data failed if the data happened to have a foreign key to another table.

Some minor enhancements were added:

  • When viewing an outline, the outline numbers are now black so they stand out from the green of the title of each entry.
  • When hoisting an outline (i.e., viewing a subtree in a large outline), we now show the complete list of the hidden ancestors so that you can just click on them to jump up in the outline.

Keyboard Shortcuts and Bug Fix Update

June 18th, 2007 by csimms

When you’re viewing a table, outline or query, there are now keyboard shortcuts to create a new item. See the list of keys here.

The fixed bug was when you had a query with a field condition with two or more negatives and no positives, the query would fail — e.g., status: not fixed, not duplicate.

Bug fix update

June 1st, 2007 by csimms

The server was updated this morning with the following bug fixes:

  • support importing OmniPlan projects (but not their native format)
  • some tasks with out of order dependencies would fail to export

The following enhancements were added:

  • export entity (e.g., task, risk, etc.) can now display graph diagram
  • all graph diagrams can be exported as Graphviz dot files — look for the “(dot file)” link

Housing Prices

April 19th, 2007 by csimms

Over the past few years, I’ve found that a suprising number of people are interested in investing in real estate. Everybody sees housing prices go up, up and up. But I just found a graph of housing prices in the US from 1890 to today, adjusted for inflation (hat tip to the Freakonomics Blog). In summary, housing prices have stayed mostly constant over the long term, once you adjust for inflation. I think people forget to adjust for inflation, and then they think that they made a lot of money by investing in real estate.

The most interesting thing to me is that over the last 10 years housing prices have doubled. That’s no surprise. What is a surprise, however, is how many people think housing prices will stay at their current levels. I think this situation is similar to the Internet stock bubble. I remember just before the bubble burst, all sorts of people who knew nothing about the stock market were putting their savings in the stock market, because they didn’t want to miss out on a sure thing. Well, that bubble burst, and I think this housing one will too, though at a slower pace since housing is less liquid than stocks.

ProjectPipe now has web services

April 5th, 2007 by csimms

We have just created a web service api for our ProjectPipe product. Our web services api follows the REST style, which means that you do queries and make changes by sending http GETs and POSTs, respectively. You verify that api calls succeeded by checking the http status code — 200 is good and 404 is an error. All output is in the JSON format, which is a simple, cross-platform data representation format with library support available for all major languages.

Below are sample URLs for project test.test:

  • /test.test/svc/task - query the task table
  • /test.test/svc/task/1 - read task with id of 1
  • /test.test/svc/task/add - add a new task
  • /test.test/svc/task/1/edit - edit task with id of 1
  • /test.test/svc/task/1/delete - delete task with id of 1

One of the great things about REST apis is that you can use any http client to use the api. For example, you can use the curl command to query for tasks:

curl -u userid:password http://localhost:8001/test.test/svc/task

Below is a python snipped to do the same thing:

import base64, pprint, simplejson, urllib2
theurl = ‘http://hosted.projectpipe.com/test.test/svc/task/1′
username = ‘userid’
password = ‘password’
encoding = base64.encodestring(’%s:%s’ % (username, password))[:-1]
req = urllib2.Request(theurl, headers={’Authorization’: ‘Basic %s’ % encoding})
out = urllib2.urlopen(req).read()
pprint.pprint(simplejson.loads(out))

If there is an error then the urlopen().read() will throw an exception, that’s the great thing about using http status codes.

Vernor Vinge’s Technology Predictions in Rainbows End

July 19th, 2006 by csimms

Vernor Vinge is one of my favorite sci-fi authors. I just read his new book, Rainbows End, and I highly recommend it. It’s set in the near term future (within 20 years), and it has a bunch of technological and social predictions I want to list here since I can’t find them on the Internet (Vinge has a 50 min podcast available which discusses some of them). These aren’t plot spoilers, but they are technology spoilers, so don’t read this if you plan on reading the book:

  • Affiliances - These are some kind of part-time work / large-scale special purpose corporations. I think the great idea here is that with the right software and network infrastructure, it should be possible to break down problems into arbitrarily small sub-problems which can be outsourced to other people anyplace in the world.
  • Belief Circles - This is directed online communities on steroids. This is when online communities which have a common belief system (say fans of Harry Potter, Trekkies, etc.) spend so much time together online that they are able to accomplish things in the real world too.
  • SHE (Secure Hardware Environment) - Following the trends of many modern governments to control encryption and other advanced or threatening technologies, the ultimate dream of government control freaks is that all computing hardware must be regulated by the government. And this government regulation requires that the lowest level computing infrastructure gives back door access to the government to everything that takes place in computers.
  • Localizers - Imagine very cheap wireless sensing machines the size of a grain of sand (something like this but smaller). And if they’re really cheap, and they’re scattered in every square meter around the country, then you’ve got ubiquitous high-bandwidth networking to everybody everywhere. But bye-bye privacy.
  • Contact lens overlays - The idea is that you can change what you see through your eyes with your contact lens (but in his podcast he says he might change it to retinal implants). Instead of painting your house, just have everyone in your family use the same color overlay!
  • SM (Silent Messaging) - People can send instant messages to others without talking or typing — I think the idea is that someone growing up using advanced computer interfaces would be able to interact with their computer interfaces without explicit or at least viewable muscle movements. Wouldn’t this just be technological telepathy?
  • Large Teams Acting as One Giant Brain - What if you had high bandwidth and productive communication between all of your team members, and the team members were experts in a large range of areas, wouldn’t the whole team acting together seem like one god-like intelligence?
  • Friends of Privacy - This is some group of unknown people on the Internet who publish subtle and not-so-subtle lies to make it difficult to trust the information you read about people online. In other words, anonymously-published large-scale disinformation.
  • YGBM (You Gotta Believe Me) - This is technology that let’s you manipulate people’s minds through broadcast media. I think it’s probably possible to control people, look at cult leaders for an example.
  • Ubiquitous Support - Imagine a real-time Google Answers. Imagine if you’re a programmer and you’ve just spent 2 days stuck on a problem, what if you could, immediately and in real-time, pay to get one hour of help from some expert programmer somewhere in the world?

How to Rotate Apache Logfiles

May 17th, 2006 by csimms

There are a number of ways to rotate your log files, and I spent a bit of time looking into all the major ones. Unfortunately, none of them satisfy all my requirements. Here are my requirements:

  • put timestamp in archive file names
  • rotate weekly (I hate the clutter of daily logfiles)
  • work with awstats and other log processing programs

I find it fascinating that even though tens or hundreds of thousands of people use the Apache web server, that a critical mass of developers have not desired what I want. I think it’s because many sysadmins who install Apache find it’s fairly easy to write a small custom script to complete the functionality they’re missing out of the box.

Let me summarize the major tools available to you right now, and then I’ll list their deficiencies.

  • logrotate: This is a standard Unix tool to rotate any application logfiles, not just Apache logfiles. It’s nice that the default Debian install sets up a logrotate job for Apache.
    • Downside: The default configuration in /etc/logrotate.d/apache2 has the entries “weekly” and “rotate 52″, which means that after 52 weeks, it will start discarding your log files! That really disturbed me, I don’t want to start losing my log files a year from now just because I forgot to move my logfiles elsewhere.
    • Downside: The naming scheme sucks: access.log, access.log.1, access.log.2.gz, access.log.3.gz, etc. And all the files get renamed during each rotation, so this isn’t compatible with rsync’ing a local copy of the files.
  • rotatelogs: This little program comes with Apache. You use it by configuring Apache to pipe its logfiles through this program. In your configuration you can specify the template file name to use, using the well-known strftime abbreviations (e.g., “/var/log/apache2/access.%Y-%m.log” would make a log file named access.2006-05.log now).
    • Downside: Not readily compatible with log processing programs like awstats, which want to read a file named access.log. You have to do extra work to make programs like awstats work.
    • Downside: Since it’s configured as a pipe, Apache actually runs it as a separate process, and this increases the amount of resources used. If you have lots of logfiles open simultaneously (say, because you have virtual domains each with their own logfiles), then you better watch out that you don’t hit a soft or hard limit on number of processes or file handles.
    • Downside: Does not compress after rotating.
  • cronolog: This is a fork of rotatelogs above, with a couple nice enhancements: 1) you can create a symbolic link access.log file which always points to the latest timestamped logfile, and 2) you can create directory structures — “/var/log/apache2/%Y/access-%M.log” would create one subdirectory for each year.
    • Downside: You can’t specify the time period for rotations, it infers it by your template file name. In the above example, it would be monthly. I wanted weekly rotations.
    • Downside: Same resource issue as rotatelogs above since it is a piped logging program.
    • Downside: Does not compress after rotating.

So what did I do? In the end, I stuck with logrotate since it was already working, and then:

  • increased number of weeks before it would delete old files to 20 years (20*52=1040). The darn program doesn’t have a setting saying “keep indefinitely”
  • wrote a little script to rename the archived logfiles to have timestamps in the file names. I can just run it in a cron job. The script just does:


cd /var/log/apache2
for i in $( find . -name "access.log.*.gz" ); do mv -i $i access.log-`date -r $i +%Y-%m-%d__%H_%M_%S`.gz; done
for i in $( find . -name "error.log.*.gz" ); do mv -i $i error.log-`date -r $i +%Y-%m-%d__%H_%M_%S`.gz; done

Hello WordPress

April 21st, 2006 by csimms

We just moved our weblogs from PyBlosxom to WordPress. I’ll subsequently explain how we did it.

Top Five Problems with Fixed Price Contracting

March 8th, 2006 by csimms

I’ve spent a number of years doing IT consulting, and I’ve been on both sides of fixed price bidding and contracts.
Below I put together five problems I’ve seen, and I describe anecdotes that really happened from the point of view of the
consulting / solution provider (I’m just hiding identifying details).
Please note that these problems are more likely to occur the larger the project is (at least $100,000).
And before you protest: some projects don’t have these problems, but most do!

Everything is Grey

No matter how detailed and specific your contract and agreed requirements are, there are always multiple ways
of interpreting human language. Unless the requirements can be evaluated by a computer, there’s going to be
a human deciding whether the requirements have been met.

Once I was a developer on a project to translate a large code base from VAX Pascal to DEC Unix C++.
My boss, not a standard IT saleman, negotiated the contract, and he was a real stickler for details.
He actually got the client (a defense contractor) to agree that we were finished development
once a set of agreed-upon unit tests passed. I think that’s pretty impressive — there could have been a million
other details needing to be translated but not covered in the unit tests (since the unit tests had to be
agreed to in advance, there was not enough time for either side to confirm that they covered sufficient ground).

Above, I was just being cheeky and showing the exception that proves the rule — almost all other projects just
have Word docs and rough prototypes as their specs.

There’s Always Contention

It’s just human nature — whenever you have people with different motivations working together, you’re
bound to get contention.

Once I was working as the technical lead on a J2EE project, and after it went live, there were performance problems.
The app ran fine under a light load,
but once a certain threshold was reached (less than 10 people!), performance dropped off a cliff. The whole team
was working really hard to figure out the problem, and I was getting pressure to go to the client’s site
to make them feel better (they were freaking out since the system was essentially worthless when it ground to a halt).
I didn’t want to fly to the client, especially since I knew that I could not figure out the solution in an airplane
or in a face-to-face meeting with an angry client trying to tell them everything was OK. We figured out the problem
(it turned out a former coworker did not know that Oracle does not automatically
create indices for you ;-) . By that time my boss’s boss was sitting on the airplane runway to fly up to the client.
Well, at least I avoided the client’s insistence to visit!

There’s Always Change

For any project that takes more than a week to complete, there will be changes, either to the specs, or to the
members of the team. Maybe the client or solution provider’s company will be bought. There can be
changes to the project’s standing in the company (maybe more important, maybe cancelled), or the project team can have greater insights while
trying to implement the project. Or the client just forgets some details. Any way you slice it, you’re going
to get change.

Once I was the technical lead on a smallish project (around 3 months with 4 people) for a pharmaceutical company,
and it was the first project by my department, so my management wanted to make sure it went well. Halfway through implementation,
the client started asking questions. How do I add new users? (Uh oh, no one spec’ed out an admin section.)
Why aren’t you following our manufacturing calendar? (Uh, no one told me about it, too bad we had tons of date-related calculations
that needed to be redone.) Where is such-and-such data? (I didn’t even know, but it turned out we needed to setup
and use a daily feed from another system.) Why isn’t your webapp following our password policies? (That’s how I learned
that we needed to integrate authentication with their Windows domain.)

Salespeople Can’t Say No

As soon as the people making the sale are different from the people doing the work, you’ve got a problem.
And the problem is that these two sets of people have different motivations. Salespeople really want
to get their commission, but they only get paid if the client is happy enough to pay the bills.
Saying no to client requests runs the risk that the client won’t pay, or will underpay, and that would hit
the salespeople’s commission. On the other hand, the actual project team
just wants to put in the minimal work necessary that they don’t get in trouble (the exception is if they get paid
for doing overtime).

Once I was stuck on a long drawn-out project, which was running late and been rescoped a couple times already.
We were doing a “final” review of the technical specs, before we were to finish implementation. In this meeting
were several representatives from the client, our developers, and representatives for upper management
from my company. At this point in the project, my management was worried that the client would bail out and not
pay. So as we went through the specs, the client was free to add any features they wanted. And did they add them!
I tried to protest, since we were supposed to be clarifying requirements and not adding them, but without any support
from my side (here my management was acting like salespeople) there was nothing I could do.

At the End, Nobody Gets What They Originally Expected

Due to all the changes that take place as a project progresses, the client loses some features they wanted, but hopefully gets
some features they needed. And the project team has to put in some overtime to get things done by deadline.

Once I was a developer on a small intranet project for a pharmaceutical, and the project team was working directly
with the business folks since the project was being run in-house. It was kind of amusing to me that the business folks
had never worked on an IT project before. Boy, did they get a surprise! They thought their project was going to be
simple and easy to implement — it was just to track salespeople’s territories. They didn’t expect that:

  • The project would be considered “validated”, thus requiring extra paperwork and slowing things down.
  • IT folks won’t do anything unless it’s been agreed to in the tiniest detail. And when the detail is not there, then you just go in feedback circles making minor tweaks.
  • Cross-system integration is painful and always takes longer than you expected — even if you already took that into account!
  • The poor business folk had to do testing because they hadn’t budgeted for external QA. How annoying!

Conclusion

Even though some things above might sound negative, all the projects completed successfully!

SVG vs. Canvas: Tastes Great, or Less Filling?

February 12th, 2006 by csimms

Now that there are two up-and-coming web graphics technologies (SVG and canvas), I thought I’d research them,
see what’s the difference between them, and see if they’re usable now.

Canvas Tag

The canvas tag got a lot of attention a while back when Apple announced they were adding it to their Safari browser,
and that they were using it to implement some of their Dashboard widgets.
What is it? Basically, the canvas tag is an “immediate mode” graphics technology — you start with an empty image jammed into your browser
page, and it’s your job to color all the pixels using javascript. It’s
going to be a “standard” by the WHATWG.

Below is a summary of current browser support for the canvas tag:

  • Firefox: 1.5 is first major Firefox release with it built-in
  • Safari: version 2 has it
  • Opera: 9 preview has it, will be in next major release
  • IE: of course it doesn’t support it. someone tried writing a small javascript emulator
    but that was just an experiment and not a real project

Here are some cool demos using the canvas tag:

SVG

SVG, or Scalable Vector Graphics, was made a standard by the W3C years ago (v1.0 in 2001 and v1.1 in 2003).
For a while it looked like just another W3C party where no one showed up.
What is it? It’s a “retained mode” graphics technology where you declare your graphical model as a tree of objects,
and the browser takes care of rendering it for you. Since the browser has your model, it can do lots of things for you like detect mouse clicks on objects and
rescale your picture to an arbitrary resolution.

Below is a summary of current browser support for SVG:

Here are some cool demos using SVG:

Conclusions

I took a graphics class in college a decade and a half ago, and the concepts remain exactly the same.
In general, SVG and canvas are complimentary, rather than competing, technologies. Anything you can do in one,
you can do in the other, but each one is optimized for certain situations.

Below, I put together a table summarizing the tradeoffs.

Feature SVG Canvas
Draw geometric shapes Easy: use SVG tags Easy: use API
Manipulate pictures Hard: Need to use poorly- or un-implemented image/feImage tags Easy: use API
Object Model Easy: You declare model, browser renders it. Model is part of DOM. Hard: No model. You can maintain your own Javascript model (fun!).
User interaction Easy: Event model is integrated Hard: None out of the box - you have to implement all event handling infrastructure, like tracking coordinates of objects in model.
Resizing Easy: that’s the S in Scalable Medium: You write the resizing code

You should know which technology to use based on your project and requirements.
If you’re just rendering some graphics, and you have no need for user interactivity, then the canvas tag is more appropriate.
If you want to have user interactivity, or if you have some other need for a model or user interaction, then SVG is more appropriate.

Ignoring the above trade-offs, if you want a technology that runs in all major browsers right now, then you have to go with SVG.
And that’s probably good, because my completely unscientific guess is that SVG is more appropriate for most (but not all!) applications.

Bottom line: SVG tastes great, but canvas is less filling.

P.S. Just for fun, I generated a GANTT chart using SVG and canvas to show how to write simple stuff
(if you have IE, don’t even try that link. :-) . I learned that for basic stuff, the APIs are quite similar. I also learned that
embedding SVG in HTML is a pain in the butt, which is why I put it in a separate page here.