Article for freelancers: The Life and Near Death of DMOZ

The Life and Near Death of DMOZ

Jim Hedger has written a widely read search marketing column
for over five years.
Co-host of Webcology on WebmasterRadio.FM www.webmasterradio.fm,
Jim is a writer and SEO consultant with Metamend Search Engine Marketing www.metamend.com in Victoria BC.
www.SiteProNews.com

This article was originally published by www.SiteProNews.com

Become a Member of TranslationDirectory.com at Just 4 EUR/Month (Paid Yearly)

Advertisements:

The casket was all but closed on the venerable Open Directory Project (ODP, or dmoz.org ). A December 16 blog post by an ODP founder, Rich Skrenta, "DMOZ had 9 lives. Used up yet?" , suggested that the directory at DMOZ is now, like Marley's ghost, deader than a doornail. DMOZ was down and, for over a month and a half, it looked like it was down for the count.

In reality, DMOZ is not dead though the rumours of its demise were not exactly exaggerated either. Because this six-week unscheduled outage followed several years of consumer dissatisfaction, lagging editorial energy, and layoffs at AOL, many made the logical assumption that the plug had been pulled.

While the website still functions as a searchable directory, its editing functions have only just been restored after six weeks of downtime. Since the last week in October, editors and submitters have been greeted by versions of a customized DMOZ 404 page. DMOZ was basically a dead directory referencing over 4 million websites spanning nearly 600,000 categories. Though editing has been restored, it is still not possible to submit new sites.

Even if webmasters could submit new sites, chances are they would not receive timely editorial attention. For the last few years, webmasters have complained about the now legendary backlog of sites awaiting review and inclusion. It can take months or even years for spelling mistakes to be corrected and an enormous number of the 590,000 categories that make up the directory do not even have editors. Though many webmasters consider the Open Directory useless because of that backlog, it still swings a big weíght in the search sector.

The greatest success of the Open Directory Project stems from the free database access offered to any other search entity. The majority of search engines and directories use the ODP's RDF-esque data-dump to help populate their databases. As every ODP listing is human edited, Google and other search engines have tended to treat ODP references as trustable sites. Carrying a PageRank of 8, links from the ODP continue to be considered Google-Gold by SEOs. Other search engines receiving results from the DMOZ directory include Ask, Yahoo and AOL. Clearly the ODP remains an important entity in the search space.

It has certainly earned its status as an important entity. The Open Directory Project has a long history that dates back to 1998. Since the day it went online as GnuHoo in June 1998 it has played a crucial, defining role in the evolution of the search sector and of the Internet.

Gnuhoo appeared on June 5 1998 in response to the rapid growth of the web. The number of new sites coming online in 1998 far exceeded the capacity of Yahoo's editorial staff that was rumoured to number less than 200. Gnuhoo co-founders Richard Skrenta and Bob Truel believed they could create a better directory using an unlimited supply of volunteer editors than Yahoo could with their limited team of professional editors.

They were right. NewHoo grew faster than Yahoo did in the last half of 1998. Less than a year after it went online, the all-volunteer project had acquired 8,000 editors and over 430,000 websites. By then it had undergone two name changes and had been acquired by one of the largest emerging online entities.

Within days of being online, Gnuhoo had attracted enough attention to force a rapid succession of name changes. First the Free Software Foundation objected to the use of the term GNU after a Slashdot article misconnected the two projects. Gnuhoo was thus renamed Newhoo. A few days later, Yahoo raised issues about the use of the suffix "Hoo". At the same time, Netscape Communications Corporation opened a dialogue with Skrenta about acquiring the upstart directory project as a major asset during their competitive phase with Microsoft.

Promising to respect the founders' original intentions to keep the site a non-commercial entity, Netscape acquired the directory for $1 million in October 1998 and renamed it the Open Directory Project. ODP data was released freely under the Open Directory Licence. A month after Netscape bought ODP, America Online (AOL) purchased Netscape. AOL agreed to honour the Open Directory Licence, formalizing it in a Social Contract with the web community. This marks the real start of the ODP's rise. By early spring 1999, most of the major search engines were pulling data from the ODP.

1998 and 1999 was a special time in the history of the Internet. Billíons of dollars were invested as eager speculators and venture capitalists moved to cäsh in on the promise of ínstant riches. Start-up companies with no functional business plans became multi-million dollar concerns overnight. The first generation of ínstant online millionaires was spawned and talk of breaking the traditional business cycle was taken seriously. The bottom was about to fall out of what had become a stratospheric marketplace but at the time, very few saw the danger through the haze of the hype. When the sky fell, it fell hard. In a tangential way, the ODP was directly involved. Though it is technically a non-profit society, ownership of the ODP is considered a business asset.

The trigger event that led to the crash of 2000 was the most significant deal in the history of global publishing. In January 2000, less than a year after it had acquired Netscape and DMOZ, AOL purchased the Time Warner media empire for approximately $160 Billion in an all-stock deal. The excess of that deal, one in which an upstart tech firm absorbed the largest brick and mortar information and entertainment business in the world, made a number of analysts look at the silliness of it all. Within three months, the shares AOL used to buy Time Warner would be worth a fraction of their value when the deal was struck.

The Tech-crash of 2000 had a cascading effect across the web. Most, if not all, of those new businesses without business plans were quickly put out of business as the value of those firms had declined and no new sources of ínvestment were forthcoming. Online properties supported by shareholders, such as Yahoo and AOL/Time Warner, were in sudden desperate trouble. 18-months of tech sector doldrums set in as the ínvestment world started looking for a revenue source that could sustain the staggering costs of the sector.

A new search engine appeared on the scene around this time. It had a funny name and appeared to disregard the dominant portal or directory structure favoured by most search engines. Hidden behind its sparse front page and childish logo was a revolutionary way of producing what everyone agreed at the time were extraordinarily accurate search results. The age of Google began in late 2000. A year later, the power of viral marketing had propelled Google into the big leagues, making it a serious challenger to AltaVista, Lycos and Yahoo.

Google populated itself in part by using DMOZ data. In its earliest years, Google used DMOZ as its directory, displaying virtually mirrored results. Google's unique method of judging page content by the number and value of incoming links made a listing at the Open Directory critically important for SEOs and webmasters. As Google's popularity and reach grew, the value of a DMOZ link grew. Because ODP listings are human reviewed, Google has traditionally tended to trust them, thus producing stronger placements faster. Between 2001 and into 2005, Google was responsible for over 80% of all organic search listings either directly or through feeding competitors such as Yahoo and MSN.

When Google figured out how to make the paid-advertising system Overture was using make oodles of monëy, all hell broke loose again and we rapidly advanced to where we are today.

When Google became the most important search engine, search marketers began targeting the Open Directory with site submissions, often with several sites for the same company. As one ODP editor put it, "We nevër asked to be used by Google like this." As the decade progressed, new methods of creating web documents (html editors, CMS, blogs, etc...) spurred another period of extraordinary growth that far surpassed the ability of DMOZ editors to keep up.

A classic dilemma existed. A link from DMOZ could mean the difference between weeks and months waiting for a good placement at Google. The ODP was nevër supposed to have such influence. The relationship Google's algorithm created between itself, the Open Directory and webmasters wanting a DMOZ listing ended up threatening the open editorial policies originally envisioned. It was difficult to enlist new editors when many applicants were primarily motivated by the ability to insert their own sites.

Though it boasts almost 75,000 editors, it also contains over 590,000 directory categories and sub-categories. The Open Directory is enormous and continues to be driven by volunteers. One of two things happen; either its volunteer editors deal with an average of 8 categories each or some categories will have to go unedited. The latter tends to happen more often than the former and the public and search engines are left with a less than complete directory to draw from. Such has been the case for the past two or three years.

In their defence, the ODP editorial staff would suggest that the majority of sites they continue to see are junk advertisement pages designed for SEO or PPC purposes. Similar comments appear in any number of threads found at the Open Directory Resource Zone, a public chat forum designed to promote communication between editors and users.

With a massive backlog of unreviewed submissions and a huge demand from search marketers hungry for the rankings boost expected from a DMOZ listing, many felt the ODP was becoming an elite, secretive society. Editorial applicants reported their requests were going unanswered and allegations of corruption amongst rouge editors emerged. By end of 2005, the ODP appeared to be in total disarray with more sites in the review process than were actually in the directory. Throughout 2006, the ODP has become less and less relevant to the search marketing community until, towards the end of the year, it was gone.

Most of the directory appears to be functioning again though it is likely a version carped together using data from the last RDF file. When the server at AOL crashed, it took most of the current directory and all of its records with it. A number of meta editors have spent the past six weeks rebuilding the directory with the help of a few friendly AOL techs. The submit a site feature is, as of this time, not functioning.

Outwardly, the importance of the Open Directory was obvious but the greatest contributions to the Internet from the Open Directory team come from the people involved with the movement and the open-source philosophy that has descended from them.

When Netscape absorbed it, the Open Directory Project became part of an amazingly influential environment. Founded by legendary Marc Andreessen, Netscape was already part of the Open Source movement. Netscape founded the Mozilla Foundation in January 1998, nearly a year before it acquired DMOZ. The Mozilla Foundation introduced and marketed the Firefox browser.

The ODP was arguably the first successful long-term project that could fall under the general heading Web2.0. Its philosophy set the stage for the Wikipedia and other community based websites. Unlike other collaborative projects that predate it, the ODP was a truly grassroots endeavour. Participants didn't need to be extraordinary technicians; they just had to be able to understand the editing techniques used by their community.

Though rumours of its death are obviously exaggerated, complaints about its demise are not. The ODP is a wonderful entity, but the power it inadvertently exerts is far greater than its ability to edit itself. Many have suggested the ODP should shut its door for good but perhaps this downtime has given its meta-editorial collective a chance to consider its role in the search community.

About The Author
Search marketing expert Jim Hedger is one of the most prolific writers in the search sector with articles appearing in numerous search related websites and newsletters, including SiteProNews, Search Engine Journal, ISEDB.com, and Search Engine Guide.

He is currently Senior Editor for the Jayde Online news sources SEO-News and SiteProNews. You can also find additional tips and news on webmaster and SEO topics by Jim at the SiteProNews blog.