- Joined
- Jul 15, 2004
- Location
- /usa/sc/florence
One of the newer teams we have started is for the Majestic 12 distributed compuing project. (http://www.majestic12.co.uk/)
What is the goal of this project?
The goal is to develop an internet search engine that is completely community supported. The Majestic 12 nodes combine machines from all around the globe to crawl, collate and then send back it's findings to the master server. The crawled data will be analysed (indexed) and added to the Majestic-12 search engine.
Can I still run other distributed computing projects?
Yes, you can configure how much CPU it uses and how much bandwidth it uses, or if you let it, it can use all your CPU and bandwidth, or anywhere in between. When archiving data barrels it uses the CPU heavily but other than that it should not use it much, unless of course you tell it to; it does use quite a bit of RAM as well.
I'm ready to get started! Now what?
This depends on your operating system. So far, it seems the windows version works much better than the Linux version, but hopefully that will be fixed in a version or two.
Tweaking the settings.
For maximun output, try some of these options:
Some screenshots are available here: http://www.majestic12.co.uk/projects/dsearch/screenshots.php
Here is a screeny from tspier2 shoing approx. system resource usage: click - and another, this one from mdameron running 170 workers.
This is something I threw together pretty quickly, so I'm sure I have some things wrong/left out, please point out some of those things for me.
What is the goal of this project?
The goal is to develop an internet search engine that is completely community supported. The Majestic 12 nodes combine machines from all around the globe to crawl, collate and then send back it's findings to the master server. The crawled data will be analysed (indexed) and added to the Majestic-12 search engine.
Can I still run other distributed computing projects?
Yes, you can configure how much CPU it uses and how much bandwidth it uses, or if you let it, it can use all your CPU and bandwidth, or anywhere in between. When archiving data barrels it uses the CPU heavily but other than that it should not use it much, unless of course you tell it to; it does use quite a bit of RAM as well.
I'm ready to get started! Now what?
This depends on your operating system. So far, it seems the windows version works much better than the Linux version, but hopefully that will be fixed in a version or two.
Windows
- Ensure that you have .NET framework 1.1 installed (link)
- Ensure you have SP1 installed for .NET (link)
- Install Majestic 12 (link)
- Fire it up
- Enter your email address, nickname, password, etc.
- If this is your first time with M12, click register. If not, double check your settings with the "Test login" button
- Go to our stats page to join our team. (click "Join this team" - link)
- You should be ready to move on to "Tweaking the settings"
Linux
- Ensure that you have Mono installed (link)
- Install Majestic 12 (link)
- Extract the package (however you like, "tar -zxvvf packagename.tar.gz" works nicely)
- Rename "sqlite3" to "libsqlite3.so.0"
- Fire it up (mono MJ12nodeMono.exe -s)
- Direct your webbrowser to http://localhost:1088/options.jhh
- Enter your email address, nickname, password, etc.
- If this is your first time with M12, click register. If not, double check your settings with the "Test login" button
- Go to our stats page to join our team. (click "Join this team" - link)
- You should be ready to move on to "Tweaking the settings"
Tweaking the settings.
For maximun output, try some of these options:
- Connection -> General: Change your connection settings, Available Downstream, Available Upstream, and the percents M12 is allowed to use
- Crawler -> Downloading: Increase Max. # of async. workers to 75-100 and Max number of open URL buckets to 10 - 15 (from the M12 site: Recommended Settings: 15 workers for 512kb-1MB line, 30 workers for 2MB line, 60 workers for 4MB line & Over 100 workers for 4MB and above. As crawler opens more buckets it will use more temporary space on your hard drive, assume that 250-300 MB will be used per bucket, but note that crawler will only open as many buckets as its necessary.)
- Misc -> External archiver: Enable using one of the external archivers for quicker uploading (under windows, try rar, if you have WinRAR put in the file path to rar.exe)
- (windows-only) Misc -> CPU Priorities / (non windows-only) Misc -> External archiver -> Process Priority: Change these according to how much or how little CPU you want M12 to use.
- (currently windows-only) Profiles: You can set up profiles to change when M12 will do work, stop doing work, etc. (For example, crawl only during nighttime hours, to stop during periods of high CPU usage, stop after a certain amount of bandwidth usage is reached, etc.) Adding profile options should be self-explanatory.
Some screenshots are available here: http://www.majestic12.co.uk/projects/dsearch/screenshots.php
Here is a screeny from tspier2 shoing approx. system resource usage: click - and another, this one from mdameron running 170 workers.
This is something I threw together pretty quickly, so I'm sure I have some things wrong/left out, please point out some of those things for me.
Last edited:
