Tracking Wikipedia
Wikipedia is one of the most fascinating resources on the web. It is too big. How do you track what goes on there? One of the best ways to track the dynamics of a wiki is to look at the recently changed pages. If it is dynamic wiki humming with activity, like Wikipedia does, this can give you some interesting clues on how busy people are, creating/updating pages on wikipedia.
So I went to look at the recent changes in Wikipedia.That page is so big, I had to set some criteria for filtering the content. I decided to track the categories of content being created. So I chose categories in a list of pull-down options and looked at the results. It was so fascinating, I decided to track it using InfoMinder. This was just a couple of days ago.
When InfoMinder tracks a page, it caches a page image and compares it with the changed page every day (or at a user specified frequency). It highlights changes for easy viewing and sends you an email aelrt. Here are a couple of fragments from the changes in “recent changes”.
Wow. There are people writing about large-scale structure of the cosmos and Half-life characters.
Wikipedia+InfoMinder combo gives me the unpredictable items of news that I will never see covered in news papers, Google news or any of the news sources. I like being surprised with categories like Fictional Extraterrestrials. Thank you Wikipedia and all those thousands of authors who keep giving us all this information. How can we ever really repay them?
Our Site Was Hacked And We Got a New Friend
It was pretty harmless affair, but a bit embarrassing. Two pages were modified with one liners against America. Thanks to Marjolein Hoekstra, who warned us in time. She first pinged me on Skype and then on Yahoo and I thought she was a spammer. Put her in the blocked list. Then she sent an email. I realized my folly, checked the site, found the problem and fixed it, immediately.
It is funny. We could have used our own tool to detect the hack. When something like this happens, you find a new use for your own tool.
Hacking incidents like this are a bit upsetting. It was a small bug in JSP Wiki, we knew about. Most of the pages on our site are wiki pages with write access to only a few. It works well for us. More about that in a later post.
Everything turns out to be good in some way. This hack caused the following event stream:
- Allowed me to get to know Marjolein Hoekstra
- Marjolein introduced me to Chris Sadds
- I found out about Touchstone (which is rather a cool service when it gets released)
- Learned about the difference between Attention Data and Attention Profile
- Learnt something I never knew about Skype
- Had a couple of nice chats with Marjolein
I love the way relationships are formed and grow in the web. Kindred spirits with similar interests find one another. In a way, I am glad we were hacked. Otherwise I would not have known about Marjolein.
Tracking Technology Trends
I am reposting this entry from my LearnLog.
If you are a developer, development manager, a software company, a tech startup or an investor, you need some way of tracking short/medium/long term trends. The Google Trends introduction last week made me think about the tools available.
This is a great tool for tracking search trends. You can type a single phrase like "venture funding" and get a trend graph. In addition, you can get information about the Top 10 cities, regions and languages for the search. You can also type multiple topics like "venture funding, bootstrapping" and see the relative trends.
2. Alexa Web Search – Top 500
Alexa is an Amazon service that tracks the popularity of web sites. Their traffic rankings method is described here.
Alexa computes traffic rankings by analyzing the Web usage of millions of Alexa Toolbar users. The information is sorted, sifted, anonymized, counted, and computed, until, finally, we get the traffic rankings shown in the Alexa service.
I mostly watch the Top-100 sites, but it may be interesting to track the Top-500. Here are a few tips on how I use it. The rankings give you a sense of who the movers and shakers are. Do a bit of extrapolation based on which industries they belong to and you get some ideas on what is going on. For example, many web 2.0 companies – myspace, facebook etc. are moving up. Alexa provides RSS feeds for many of its services.
3. Blog Trends with Technorati
Type a search term, say "Java" in the search link. Then click "more" below the chart displayed on the left of the page. You will get to a page like this. Now you can refine the search a bit using the various options in "authority" and blog type. This is a pretty neat tool. Till I tried it out, I did not even know that bloggers are ranked by authority.
Technorati also publishes some interesting white papers on the emerging trends on use of blogs.
4. PubSub
Another very useful resource is PubSub. You can subscribe to keywords/phrases and receive alerts. My favorite part of this site is PubStats.
PubSub monitors millions of feeds. By generating a list of all the URLs contained in entries of each feed, it's possible to determine a site's relevance just from the number of incoming links it has. LinkRank goes one step further and calculates a score for each linking site. Sites are then scored based on the score of the sites that link to them.
5. Del.icio.us
Del.icio.us is a collaborative bookmarking service. It allows people to store their bookmarks and tag them. As a user, you can retrieve your own bookmarks, popular ones and others' by using tags. The level of activity and popularity provide you with an idea on some of the short term trends.
6. Diggdot.us
Diggdot.us combines Digg.com, Slashdot and del.icio.us to provide an integrated site for tracking the most popular technology related posts.
7. Tag Clouds
Over the past year or so, tagging is gaining popularity. Several services including Yahoo, Technorati provide tag clouds. A tag cloud is a list of tags displayed in a box. The tags used more heavily have bigger fonts. Here is an a tag cloud on Web 2.0 from technorati.
Tag clouds provide great visualization for popular tags. There are tag cloud animations that show you how the tag popularity increases over a period of time. Here is one of the most popular tag cloud animations from Jon Udell.
There are various other methods:
- Tracking mashup activity,
- Following TechCrunch and eHub,
- Tracking booklists,
- Using Google/Yahoo alerts,
- Watching user group activity
Ultimately the velocity of information in the media will give you a pretty good idea about short term trends.
Tech Mining
This is a re-post from my LearnLog. I have been a reading a book called Tech Mining. I was planning to write a few blogs after finishing the book. But the whole purpose of my learn log is to (b)log as I learn. So here some information from the first couple of chapters.
According to the authors, various types of Technology Analyses can be aided by tech mining.
1. Technology Monitoring(also known as technology watch or environmental scanning) – cataloguing, characterizing, and interpreting technology development activities.
2. Competitive Technology Intelligence(CTI) – finding out "Who is doing what?"
3. Technology Forecasting – anticipating possible future development path for particular technologies
4. Technology Roadmapping – tracking evolutionary steps in related technologies and, sometimes, product families.
5. Technology Assessment – anticipating the possible, unintended, indirect, and delayed consequences of particular technology changes.
6. Technology Foresight – startegic planning(especially national) with emphasis on technology roles and priorities
7. Technology Process Managment – getting people making decisions about technology
8. Science and Technology Indicators – time series that track advances in national (or other) technological capabilities.
We do a bit of the first activity with our product InfoMinder, but have a long way to go in provide the other capabilities mentioned above. We do plan to help customers set up Information Portals to store the tracked information and do some automatic linking.
Companies like Techdirt do a great job of Tech Mining and provide it as a service to customers. You can find the Techdirt Blog here.
What is InfoMinder?
This blog is about InfoMinder, a product from our company iMorph, Inc. We would like to bring you information on how:
- To keep track of blogs
- To track web-sites and receive notifications when they change
- To track specific pages in a wiki
How do people use InfoMinder?
Each user has a different interest. Here are some of the things they track:
- Competitor sites for new product introduction, partnerships and customer wins
- Customer sites to stay in touch with customer news and information
- Government sites for local and Federal Information
- Immigration sites for status changes
- Compliance related information
- To track favorite blogs
- Public knowledge bases
- Internet Research
- Competitive Intelligence
- Lead generation
- Tracking trends
- To track changes to wikis (which are becoming community knowledge portals)
- To be alerted when your site is tampered with
And many more applications. In future entries in this blog, we will cover how you can use simple tools like InfoMinder to stay aware of what is happening in your industry or any area of interest.
You can try InfoMinder free for one month here.
