Gentoo Weekly Newsletter: December 5th, 2005The compiler is a central component of every Gentoo system, so unlike other packages, the developers are rather careful when updating it. Finally gcc 3.4.4 has made it to the stable branch on x86, affecting a large number of users. The announcement on the developer mailing list links to the upgrade guide containing the steps necessary for the upgrade. It is important to rebuild some packages before unmerging the old gcc version or the system may break. According to the thread in the Gentoo forums, some minor problems occured on some systems due to a small bug, but once it was fixed the upgrade worked fine for most of the users.
Power.org opens free developer membership Gentoo PPC developer Luca Barbato was the first to hop aboard a new initiative from Power.org, a collaborative industry consortium dedicated to "developing, enabling and promoting Power Architecture technology innovation" and representing a wide range of semiconductor and electronics organisations active in the field. Founded by IBM, Power.org tries to open itself to the community of POWER, PowerPC and related architecture development, and counts Gentoo sponsor Genesi among its recent additions to contributors and supporters. Developers who become members of Power.org get access to technical resources at the member-only area on the website, and to technical subcommittee output. You can register at the Power.org website. Looking at this week's unusually rich issue of the GWN, you wouldn't suspect that we sometimes have serious trouble getting our weekly edition out in a timely and accurate fashion. The author list on this page has dwindled down to less than a handful of regular contributors. If you would like the GWN to continue to provide you with information from the Gentoo project, all its outer layers in the user communities, commercial spin-offs or fun stuff emerging from the greater Gentoo realm -- then we'd like to encourage you to join us on the authoring team. Many of our current news sections could use a helping hand, for example everybody's favorite GWN chapter, the Tips & Tricks, is particularly in need of input from as many sources as possible. Send a message to gwn-feedback@gentoo.org and tell us for which of the existing sections you'd like to contribute articles, or maybe come up with entirely new ideas, offer proofreading (very much appreciated), invent new and exciting techniques to automate some of our scripting, or indeed whatever else you feel up to. 100k users registered at the Gentoo Forums Three and a half years after the creation of the Gentoo forums we hit 100,000 registered users early on 16 November morning when Blindside350 registered:
A forum thread from a very distant past has been reanimated for the occasion. Interview with Bryan Green at NASA's Advanced Supercomputing division
Tell us about yourself and your work with Columbia (along with a few words about Columbia for those who are not familiar) I am a contractor working at the NASA Advanced Supercomputing (the NAS, formerly "Numerical Aerospace Simulation") facility, at NASA Ames Research Center in Mountain View, CA. I am an employee of AMTI. I've been a member of the NAS facility's scientific visualization group since 1997, when I graduated from UC Santa Cruz. When I started, we were using SGI Onyx systems for visualization, but the emergence of NVIDIA, and the pressure of the PC gamer market on the graphics card industry, ultimately made it possible to move to commodity-based linux systems with high-end NVIDIA graphics cards. So we have the gamers to thank for saving us hundreds of thousands of dollars on graphics hardware. My group's home page is here: http://www.nas.nasa.gov/Resources/Visualization/visualization.html The Columbia supercomputer, a 10,240 processor SGI Altix system running Linux, was built by the NAS in 2004. It uses Intel Itanium2 processors, and is divided into 20 nodes with 512 processors each. Each 512-processor Altix node is a Single System Image (SSI) with 1 Terabyte of shared memory. More information about Columbia and the various types of science being performed on it can be found here: http://www.nas.nasa.gov/About/Projects/Columbia/columbia.html While Columbia itself uses a Suse-based distribution (with SGI modifications), the NAS division's security group uses Gentoo exclusively for infrastructure and special projects. They even have a three-node Mosix cluster. In total, they have about 30 Gentoo systems deployed. If anyone is interested in more information about this, they can contact Derek Shaw. Since Columbia was built, my group's work has primarily involved providing visualization support to Columbia users. Our focus these days is on tapping directly into the simulation codes themselves, and extracting data as it is produced. We have our own 16-cpu-node "chunnel" plugged into Columbia. For each simulation timestep, we extract the computed data via shared memory, ship the data over infiniband to chunnel, and distribute it to our visualization cluster. During this past hurricane season, we have been using this technique to generate movies from runs of fvGCM, the finite-volume General Circulation Model which has been used in hurricane forecasting. This was one of the applications we were demonstrating at SC05. On to the Columbia monitor. What all are you able to visualize and see on these screens? What else do you do with this visualization cluster? We refer to our grid of LCD screens backed by a cluster as a "hyperwall". As opposed to a "powerwall", which typically displays large images, our intent is to visualize high-dimensional datasets by allowing the use of different tools, viewpoints, and parameters to display different aspects of a dataset simultaneously and in concert. We have a 7x7 hyperwall installed at our facility and a 3x3 hyperwall which serves as our "portable": the one you see in the picture. The hyperwall has been used for a number of applications including Computation Fluid Dynamics (CFD), quantum physics, astronomy and astrophysics, and atmospheric and oceanic modeling. The 7x7 is still running Fedora Core 2. It is our original hyperwall, and whether/when/how it gets updated and/or converted is subject to debate. The Columbia monitoring tool is a program for monitoring system resource utilization on one or more Columbia nodes. With the hyperwall, we can monitor all 20 Columbia nodes simultaneously with plenty of resolution. Our ultimate goal is to combine system resource monitoring with code instrumentation to get a holistic view of a specific simulation's behaviour, performance and system utilization, ultimately for optimization purposes. Our division provides "full-service" support for High End Computing (HEC) applications -- porting and scaling, optimization, and visualization.
Most of the statistical information is gleaned from the proc and sysfs filesystems. Three monitoring components are shown in the picture: cpumon, pagemon, and linkmon. Cpumon visualizes cpu statistics gleaned from '/proc/stat', as well as showing cpuset allocations. Pagemon visualizes the non-uniform memory usage -- memory in a 512-processor node is divided into 256 blocks with 4 Gigabytes each, with statistics gleaned from '/sys/devices/system/node/node*/meminfo'. Linkmon uses the "Performance Co-Pilot" tool to get NUMAlink statistics -- Columbia's communication backbone. The charts in the upper left screen show cpu and memory statistics for the 64-processor Columbia frontend node (bottom), and the vis group's 16-processor node previously mentioned (top). Each of the 5 screens arranged in a cross is subdivided into quadrants, with each quadrant showing statistics for a single Columbia 512-processor node. A set of unhappy faces means either a node is down, or the monitoring daemon is not running on that system. The set of three views on the lower right screen is just a blowup of the Columbia node on which the production fvGCM runs are performed, from which we produce our atmospheric visualizations. What do the colored boxes, bar graphs, and lines represent? The grid of colored boxes (cells) shows cpu utilization. Each cell represents one cpu. The color codes are as follows: black=idle, blue=user, red=system, cyan=iowait, yellow=soft-interrupt, orange=hard-interrupt. Cpusets are represented by colored outlines around groups of cells. The red bar graphs show memory usage. For a 512, there are 256 vertical bars representing each of the 4-Gigabyte blocks of memory. The height of the red bar indicates the percentage of memory allocated. The line graph shown below the cpu and memory charts is a representation of the NUMAlink topology for a 512-cpu node. To quote our SC05 poster: "Dual-plane fat tree NUMAlink interconnect topology, with activity shown in magenta between the 2 nodes within the green squares in a C-brick, and between the nodes and the routers represented by cyan squares; level 1 and level 2 metarouters are indicated by the yellow squares." Note that the term "node" here refers to a portion of a C-brick, not to an entire 512-cpu "node". Magenta=activity, blue=inactivity, gray=unreadable. Give us a few details about how the 3x3 screen wall works. How do you control it? The hyperwall is a fairly simple cluster. Each node is a dual-CPU, dual-core Opteron system with 4 Gigs RAM. Each node has a basic Gentoo install on it, with Xorg and Xfce, and of course the NVIDIA video driver for OpenGL. We use NFS for shared filesystems. We have a set of scripts for things like launching programs on the nodes. Nodes can be selected by their screen's two dimensional position in the grid, using perl regular expressions to select groups of nodes -- so the cross pattern used in the picture is described by: 'CROSS = N1. N.1'. We have a tool called 'hyperX' which allows you to drive the cursor on all the screens, or subsets of screens, simultaneously from the master console. It works using the XTest protocol. I have also written a C++-based distributed object and event architecture that is used for driving some of the applications on the wall. This is the underlying framework used by the Columbia monitoring tools.
One application approach is to launch a copy of an application on each screen and drive each instance simultaneously. Each instance of the app has different initial conditions, for example the data field being mapped onto a particular geometry, or a variation in the colormap. The apps are driven either via their GUI, using hyperX, or via a master control panel that controls the apps with app-specific protocols. The advantage of hyperX is that virtually any application can be used unmodified. Another approach involves more intercommunication between the apps running on the nodes, so that the view on one node influences the view on another in some way. For example, constraints in one set of dimensions may be set visually in one display, and the mapping of those constraints into other dimensions may be seen in the other displays. Why did you pick Gentoo for this setup? I simply wanted to try installing Gentoo on a cluster. I first installed Gentoo on my work laptop, and after becoming familiar and comfortable with it, I installed it on my workstation. Until then, my group's systems were exclusively Redhat/Fedora. Gentoo reignited my love for Linux, and it made system administration interesting and enjoyable for me. I started getting the idea that it would make a great distribution for a cluster, and when we built our first mini hyperwall (the first 3x3), I had partitions set aside for later installing Gentoo alongside Fedora. After a good long wait, I finally did the deed, and it actually went really well. I then installed Gentoo on our second 3x3 about a week before SC05... well, actually, I cloned the original 3x3, and then just tweaked and updated. If you want a rational motivation for the install, besides simple experimentation, it was because I really like and depend on Portage for convenient access to a diversity of applications and keeping certain packages up-to-date, like GTK. I was always having trouble with installing non-standard packages on Fedora, usually because no one had bothered to make an RPM for them, or I got stuck wrangling with yum. What are the big advantages of using Gentoo in this particular setup? How about the disadvantages or areas that need improvement? For one thing, the ability to find useful information, documentation, and community support is very valuable and something that I always felt was lacking from Redhat/Fedora. I setup my first Portage overlay to deal with an issue involving our newest 3x3. The NVIDIA video driver had major stability issues on our hardware. We got NVIDIA to identify and fix the problem, and they provided us with a patch. I copied the latest nvidia-kernel ebuild into a Portage overlay and updated it to apply the patch. Its great to be able to have our custom-patched graphics driver reside in the package management system -- and it was really easy to do. Portage offers a number of advantages over binary distributions. Access to a diversity of packages is one. Even browsing/searching Portage is a great way to learn about what tools are out there. The ability to be selective about what versions of packages get installed has come in handly as well. Another advantage is the ability to incrementally update a system, or parts of a system, from the command line. In fact, having all system administration tasks be commandline and script-oriented is a real plus on a cluster, in my opinion. The flexibility of Portage/Gentoo is one of the reasons for using it on a cluster. My current setup has the Portage tree being shared over NFS to the nodes, with squid on the master providing a proxy for emerge. I also have some homebrew scripts for cloning using rsync. I have other configurations in mind to try, like installing the node OS in a chroot environment on the master, and just propagating it from there to the nodes. Making Xfce the desktop manager on the nodes has also been a big improvement over using Gnome (on Fedora), which was just massive overkill. One thing I think Gentoo needs is its own version of the SystemImager tool. A 'GentooImager' ebuild, if you will. SystemImager is tailored to Redhat/Fedora, and includes the whole kitchen sink in its distribution. A GentooImager ebuild, perhaps based on and borrowing from SystemImager, could just make the necessary tools -- dhcp, syslinux, tftp, etc. -- be dependencies. Beyond that, there's the matter of providing an environment for pxeboot to boot into and bootstrap a system and the other details of making system images. Anyway, its something to think about. When/if we build another hyperwall, I may start working on such a package. We'll see. I'm interested in hearing from others who have installed and maintained Gentoo on a cluster. As for disadvantages, I don't see any particular disadvantages to using Gentoo on the hyperwall, though it might be difficult coming at it as your first Gentoo install. Areas of Gentoo that I see as needing improvement are in handling security-specific updates, and in tracking important messages that are printed during an emerge. I'm glad that these are issues that appear to be getting attention. Are there any future plans for Gentoo installations in your department? Right now, I'm just the annoying Gentoo flag-waver of my small group. I've managed to generate some interest among a portion of my group, and I think one of them is going to give Gentoo a try. We'll see what happens. As I mentioned, Gentoo plays a major role in the security department. As for Columbia itself, well, one can dream... How can you be contacted? Contact information can be found via my web page contents of /dev after initial installation The new dynamic device managers (deprecated devfs and udev) handle all device nodes dynamically. So why are there lots of static device nodes in /dev? Gentoo can boot without them, but for system rescue it is a lot more convenient to have a fallback. You could remove all but one or two static device nodes and still boot, only debugging would be difficult as you might not have terminal output ... India: FOSS.IN conference with Gentoo participation Last week India saw its largest free and open-source software conference -- Foss.in/2005 held at the Bangalore Palace. Gentoo was officially represented by developers Seemant Kulleen and Shyam Mani, in addition to local enthusiasts Ashish V, Ananth, and Arun Raghavan.
The first appearance of Gentoo at the event occured in the form of all the conference's servers and majority of the desktops running Gentoo. The Gentoo stall that was set up in the expo area was rarely without a crowd of both Gentoo and non-Gentoo users. Much interest was shown in how Gentoo is different from other distributions and existing users were assisted with troubleshooting their issues. More than 100 CDs (installation and package) were burnt over a period of 2 days. The afternoon of Day 2 saw an entire 250-seater hall dedicated to everything Gentoo. Seemant kicked the session off with an introduction to the history of Gentoo, and a brief run-down of what makes Gentoo tick. Amidst much applause, Seemant handed the stage to Shyam who, after dispelling the "Gentoo is for Ricers" line of thought, walked through the various features of Gentoo that users would need to be aware of. Arun followed this up with a quick Ebuild 101 session. Seemant returned to present a number of case-studies of Gentoo in production environments. Each talk ended with a number of questions from the audience. The session was ended with an open-ended QA session and a quick demonstration of how to write an ebuild. Though there were many requests to demonstrate an actual installation, users were encouraged to try this themselves, then seek assistance at the Gentoo stall if required. In a short session with the localisation folks, an rough action plan to include localisation and Indic font support was drawn up.There was also talk of setting up a local Gentoo users group. Watch out for this in coming months! The team also wishes to thank Harikrishnan for coming up with some awesome posters that were made use of in the Gentoo Stall. Germany: "Friends of Gentoo e.V." at the "Linuxtage in Essen" Last weekend the German not-for-profit organization Friends of Gentoo e.V. managed to set up a booth at the first Linuxtage in Essen. Besides the usual things like demonstrating Gentoo on several notebooks and more obscure hardware (this time an old Compaq desktop machine) the booth team sold a special and limited printed edition of the German Gentoo Handbook. The first day of this new exhibition on the open-source conference trail exceeded expectations and was a huge success, with more than 1300 visitors. The second day, Sunday, was topped by a presentation about the Gentoo Metadistribution, by Gentoo developer Tobias Scherbaum. Retro Gaming Hacks (October 2005) A new book from O'Reilly, Retro Gaming Hacks, is dedicated to helping 20- and 30-something gamers get their nostalgia on (while introducing younger gamers to those gems of our gaming past). In addition to actually running retro games on real retro hardware, the book gives readers the know-how to emulate classic gaming systems on modern computers. Yes, even modern computers running UNIX operating systems. And that, of course, is where Gentoo enters the picture. Hacks #22 (Play Arcade Games Under Linux), #40 (Run NES Emulators Under Linux, #43 (Emulate Other Class Systems in Linux), #51 (Use Console Controllers on Your PC), and #52 (Use USB Gamepads Under Linux) were written by Gentoo developer Josh Glover, who makes sure to point out how easy it is to install emulators in Gentoo Linux as opposed to other distros. Retro Gaming Hacks should be available from finer booksellers everywhere; or read it online using O'Reilly's Safari Bookshelf. The following developers recently left the Gentoo project:
The following developers recently joined the Gentoo project:
The following developers recently changed roles within the Gentoo project:
A vulnerability has been identified that allows a specially crafted SVG file to exploit a buffer overflow and potentially execute arbitrary code when opened. For more information, please see the GLSA Announcement chmlib, KchmViewer: Stack-based buffer overflow chmlib and KchmViewer contain a buffer overflow vulnerability which may lead to the execution of arbitrary code. For more information, please see the GLSA Announcement The Gentoo community uses Bugzilla (bugs.gentoo.org) to record and track bugs, notifications, suggestions and other interactions with the development team. Between 28 November 2005 and 05 December 2005, activity on the site has resulted in:
Of the 9129 currently open bugs: 97 are labeled 'blocker', 207 are labeled 'critical', and 560 are labeled 'major'. The developers and teams who have closed the most bugs during this period are:
The developers and teams who have been assigned the most new bugs during this period are:
Please send us your feedback and help make the GWN better. 10. GWN subscription information To subscribe to the Gentoo Weekly Newsletter, send a blank email to gentoo-gwn+subscribe@gentoo.org. To unsubscribe to the Gentoo Weekly Newsletter, send a blank email to gentoo-gwn+unsubscribe@gentoo.org from the email address you are subscribed under. The Gentoo Weekly Newsletter is also available in the following languages:
|
|
