The gentoo.org redesign, Part 4: A site reborn

1.  The final touch

A new look, but...

At the end of the previous article, the Gentoo Linux Web site had a completely new look, but there are still some things that aren't quite complete. In this article, the final installment in this series, I finally put those finishing touches on the site, resulting in a fully-functional, refined, and modular XML-based site that's ready for the future. Here's what was missing from the site since the last article:

Loose ends

First, while the site has a completely new look, only the documentation portion of the site is XML-based. The main "category" pages are still in raw HTML and need to be converted to an XML/XSLT solution to make things more maintainable and expandable.

Also, my developers have found several problems with the raw HTML itself. The site looks particularly bad when viewed under Netscape 4.77 -- obviously, this is a problem. Also, there are a number of other minor rendering problems that appear in more modern browsers, the most annoying of which is a thin vertical black line that does not extend completely down the entire page, ruining the illusion that the main content area is being spoken by our flying-saucer guy. Also, our documentation pages don't completely match the more refined look of our new main category pages -- clearly something worth updating.

The goal

Here's the plan for the final rework of the Gentoo Linux site. First, we'll totally rework the main page HTML, keeping the same overall look, but making the page more browser-compatible. At the same time, we'll add a few presentation-related refinements suggested by our visitors, and also fix browser compatibility problems with our existing "guide" documentation system.

Next, we'll completely move the site over to XML and XSLT. By the end of this article, any change made to the site will be made by modifying XML or XSLT rather than directly editing HTML, which will now be generated automatically with the help of xsltproc. This will make the site a whole lot easier to maintain. Because Gentoo Linux is a community-developed project, this will, in turn, allow our developers (and me) to maintain and improve the site as needed. I'm really excited about this since it will save us a bunch of time and ensure that our visitors are greeted with up-to-date content.

Compatibility issues

While Netscape 4.x is still a very widely used browser, it is difficult for me to decide exactly how many hoops to jump through in order to make the site look better when viewed through this browser. Should I merely ensure that the site is readable (without any major glitches) or should I do everything I can to make sure the site looks absolutely perfect under Netscape 4.x, even if that means using less or no CSS and adding strange compatibility hacks to the existing HTML?

In the end, I decide to make several major changes to the HTML so that the site will still look quite good under Netscape 4.x without focusing too much on minor bug-related table spacing and font-rendering issues. Here are some of the changes made to the site's HTML to get everything 4.x compatible. (The Gentoo Linux development team has submitted several of these fixes.)

First, Netscape 4.x has a bug that causes CSS background colors of block elements to be displayed incorrectly. For example, here's how a particular portion of a guide document is supposed to be rendered:

Figure 1.1: A sample guide document in IE5

And, here is how Netscape 4.x renders this same portion when background colors are specified using CSS:

Figure 1.2: A sample guide document in Netscape 4.7; some fixes are needed

This is ugly. To fix it, existing block-level elements, such as this one...

<p class="note">This paragraph doesn't look so good in 4.x</p>

...were replaced with tables, as follows:

<table width="100%" border="0" cellpadding="0" cellspacing="0">
        <tr>
                <td bgcolor="#ddffff"><p class="note">
                This looks a whole lot better in 4.x</p></td>
        </tr>
</table>

This hack fixes the background-rendering problem. However, this "fix" also requires color information to be included in the HTML, which undermines the benefits of using CSS in the first place. This is an unfortunate situation, especially for fans of CSS like myself, but is required for Netscape 4.x compatibility.

Rebuilding the HTML

Now it's time to deal with the black vertical line that doesn't always extend all the way to the bottom of the screen. I have been unable to find a solution to this problem that works in both a 4.x and 5.x browser; every 5.x version has triggered bugs in Netscape 4.x, and every 4.x-compatible version looks horrible in a 5.x browser. So, I decide to simply remove the black line entirely: Finally, the site works in all popular browsers. Next, I will to create a guide-like syntax for creating the main pages.

Approaching the XML

Instead of implementing a completely new tagset for the main page, I think it would be a good idea to try to use as many of the "guide" XML documentation tags as possible (see part 2 of this series for more information on the guide XML format). So, I hack away at some new XSL, using my guide XSL as a template for my work. After an hour or two, I have a fully-functional set of XSL transformations for turning a guide-like syntax into an HTML main page. Revision 2 of the new main page looks like this:

Figure 1.3: The new main page revision

Now that the main page is using a new XML/XSLT backend, I direct my attention to the "guide" system's HTML output. Not only do I need to fix a host of Netscape 4.7 compatibility bugs, but I also need to further update the generated HTML and graphics so that they will match those of my newly-revised main page. Then the idea strikes me: Why not simply tweak my new main page XML/XSL just a little bit so that it can also generate HTML for my documentation? After all, I have just added support for nearly every "guide" XML tag, so that they can also be used for main page content.

This solution turns out to be really easy to implement. I just tweak the new XSLT file so that it will remove the left-hand "link bar" and perform a few other minor changes to the output HTML when it processes documentation pages. Since most of the XSLT is still the same, I can use a single set of master XSLT templates for both the guide documentation and the category pages:

Figure 1.4: How the new XSL works

Not only do I now have a single set of XSLT templates to maintain, but because both flavors of output HTML are based on the same master document, they now share the same CSS stylesheet. This means that there is no need to "synchronize the look" between two disparate sets of stylesheets and output HTML. And as you can see, the new documentation HTML is a perfect match for the new main page:

Figure 1.5: The new documentation pages perfectly match the new main page

The XML implementation

The actual implementation is quite easy; my existing guide XML syntax requires that every document be part of a single master <guide> element. To add support for main category pages, I create a new master element: <mainpage>. To create a main category page, I place everything inside a <mainpage> element instead of a <guide> element, and the XSLT makes the appropriate changes to the output. Besides this, the only major change required is the addition of an optional <sidebar> element that's used to specify the contents of the floating table on a main category page. The existing <guide> XSLT template looks something like this:

<xsl:template match="/guide">
        <html>
        <head>
                guide header goes here
        </head>
        <body>
                top part of guide body HTML content goes here
<!--next, we insert our content-->
                <xsl:apply-templates select="chapter" />
                bottom part of guide body HTML content goes here
        </body>
        </html>
</xsl:template>

If you're not too familiar with XSLT, this template tells an XSLT processor to replace the <guide> </guide> tags with the shell of an HTML document, as well as recursively applying templates to any <chapter> elements (opening/closing tag pairs) inside the <guide> element and inserting the resultant output into the middle of the HTML shell.

So, to add support for the main category pages, I need to specify that a different HTML shell should be used if everything happens to be enclosed in a single <mainpage> element. To do this, I add a new template, as follows:

<xsl:template match="/mainpage">
        <html>
        <head>
                mainpage header goes here
        </head>
        <body>
                top part of mainpage body HTML content goes here
<!--next, we insert our content-->
                <xsl:apply-templates select="chapter" />
                bottom part of mainpage body HTML content goes here
        </body>
        </html>
</xsl:template>

Because nearly every other XML element (from <chapter> all the way on down) produces identical HTML output for both guide and main category pages, almost every other XSLT template can be shared for both types of pages. Thus, we can get along just fine with a single XSLT file that specifies two "HTML shells" and a common set of XML-to-HTML XSLT templates. As always, code reuse is definitely a good thing.

The Changelog page

You'll remember that in Part 2 of this series I mentioned that the cvs2cl.pl CVS Changelog generation script could produce XML output and that I wanted to eventually use this feature as the basis for a daily CVS Changelog page that would appear on the new Web site. Now, with the new XML backend in place, adding the new Changelog page is a piece of cake. Here's an enhanced version of the cvslog.sh script that also takes care of handling the XML-to-HTML conversion:

#!/bin/bash
#various paths
HOMEDIR=/home/drobbins
CVSDIR=${HOMEDIR}/gentoo/gentoo-x86
OUTLOG=${HOMEDIR}/gentoo/xmlcvslog.txt
OUTMAIL=${HOMEDIR}/gentoo/cvslog.txt
WEBDIR=/usr/local/httpd/htdocs
XSLTP=/opt/gnome/bin/xsltproc
TMPFILE=${HOMEDIR}/gentoo/xmlcvslog.tmp
USER=drobbins
#if $CVSMAIL is undefined, set it to "yes"
if [ -z "$CVSMAIL" ]
then
        export CVSMAIL="yes"
fi
#the main script
cd $CVSDIR 
cvs -q update -dP
yesterday=`date -d "1 day ago 00:00" -R`
today=`date -d "00:00" -R`
cvsdate=-d\'${yesterday}\<${today}\'
nicedate=`date -d yesterday +"%d %b %Y %Z (%z)"`
#generate cvs2cl.pl XML output
/usr/bin/cvs2cl.pl --xml -f $OUTLOG -l "${cvsdate}" 
#use sed to remove "xmlns=" from cvs2cl.pl output
/usr/bin/sed -e 's/xmlns=".*"//' $OUTLOG > ${OUTLOG}.2
#convert cvs2cl.pl XML output to guide format using $XLSTP
$XSLTP ${WEBDIR}/xsl/cvs.xsl ${OUTLOG}.2 > $TMPFILE
#convert guide XML output to HTML format using $XLSTP
$XSLTP ${WEBDIR}/xsl/guide-main.xsl 
$TMPFILE > ${WEBDIR}/index-changelog.html
#fix perms
chmod 0644 ${WEBDIR}/index-changelog.html
#automatically send cvs mail if $CVSMAIL is set to "yes"
if [ "$CVSMAIL" = "yes" ]
then
        /usr/bin/cvs2cl.pl -f ${OUTMAIL} -l "${cvsdate}" 
        mutt -x gentoo-cvs -s "cvs log for $nicedate" > ${OUTMAIL} 
fi

While this script may look significantly more complicated than the earlier version, it really only contains four or five key additional lines; the rest of the additions are either comments or environment variable definitions.

Here's how the new XML-related parts of the cvslog.sh script work. First, we call cvs2cl.pl and instruct it to generate an XML-based Changelog containing all the files that were modified yesterday. Then, this XML output is run through sed to remove an unneeded xmlns= attribute from the XML. Next, we hand this slightly tweaked XML over to xsltproc and tell it to apply the processing found in cvs.xsl; these instructions transform the XML output from cvs2cl.pl's into a proper guide XML document. Finally, we again use xsltproc to convert this guide XML document into Web-ready HTML, which is piped into our Web server's htdocs directory. The generated HTML Changelog page is complete, and this is the result:

Figure 1.6: The automatically generated Changelog page

You might be surprised at the simplicity of the XSLT contained in cvs.xsl. In it, we specify three templates for <changelog>, <entry>, and <file>. We also make reference to a few other tags in the source XML, including <date>, <author>, and <msg> (which cvs2cl.pl uses to specify the CVS committer's comments). cvs.xsl does quite a bit considering that it is only around 35 lines long:

<?xml version='1.0' encoding="iso-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output encoding="iso-8859-1" method="xml" indent="yes"/>
<xsl:template match="/changelog">
        <mainpage id="changelog">
        <title>Gentoo Linux Development Changelog for <xsl:value-of select="entry/date"/></title>
        <author title="script">cvs-xml.xsl</author>
        <standout>
                <title>About the Development Changelog</title>
                <body>
                        This page contains a daily Changelog, listing all modifications made to our
                        CVS tree on <xsl:value-of select="entry/date"/> (yesterday).
                </body>
        </standout>
        <version>1.0.0</version>
        <date><xsl:value-of select="entry/date"/></date>
        <chapter>
                <xsl:apply-templates select="entry"/>
        </chapter>
        </mainpage>
</xsl:template>
<xsl:template match="entry">
        <section>
                <title>Files modified by <xsl:value-of select="author"/> at 
                                        <xsl:value-of select="time"/>
                </title>
                <body>
                        <note><xsl:value-of select="msg"/></note>
                        <ul>
                                <xsl:apply-templates select="file"/>
                        </ul>
                </body>
        </section>
</xsl:template>
<xsl:template match="file">
        <li><path><xsl:value-of select="name"/></path>, <xsl:value-of select="revision"/></li>
</xsl:template>
</xsl:stylesheet>

Project complete!

Since the beginning of the Gentoo Linux Web site redesign, we've created a user-centric action plan, designed a new XML-based documentation system, a new logo, a new look for the site, converted all remaining parts to XML, and added a new XML-based Changelog page. Phew! I hope that you've enjoyed following my progress, and have found ample ideas and inspiration along the way. I've received several requests for more information and code related to the redesign, so I've set up a special Gentoo Linux XML Projects page that contains the most recent XML, XSLT, scripts, and documentation used for www.gentoo.org. In addition to visiting the Projects page, be sure to check out the valuable resources listed below.

2.  Resources

• Check out the Gentoo Linux XML Projects page if you're interested in using the Gentoo Linux guide XML system as a basis for your own projects. All the latest XML/XSLT stuff can be found here.
• Read the other articles in this developerWorks series about the redesign of the www.gentoo.org Web site using technologies like XML, XSLT, and Python:
  • In Part 1, the author creates a user-centric action plan and introduces pytext, an embedded Python interpreter (March 2001).
  • In Part 2, Daniel shows off the new documentation system and sets up a daily CVS-log mailing list (May 2001).
  • In Part 3, he creates a new look for the site (July 2001)
• Learn more about CSS (cascading style sheets) by visiting the CSS page at the World Wide Web Consortium, or W3C. You can find out more about XML, XSLT and a host of other technologies as well.
• Check out Xara.com, the home of Xara X -- an excellent vector drawing package for Windows. With virtually no bloat and blazing speed, it has my personal recommendation.
• Learn even more about XSLT at http://www.xslt.com
• When you wake up, check out Sablotron, a nice, fast XSLT processor available from http://www.gingerall.com