Introduction to RAID

RAID is one of those technologies that has really revolutionized storage. In this article, we’ll review the six most common single RAID levels and describe how each works and what issues surround them.

Introduction
One of the most common techniques to improve either data reliability or data performance (or both) is called RAID (Redundant Array of Inexpensive Disks).

The concept was developed in 1977 by David Patterson, Garth Gibson, and Randy Katz as a way to use several inexpensive disks to create a single disk from the perspective of the OS while also achieving enhanced reliability or performance or both.


Before anyone erupts and says that RAID does not stand for “Redundant Array of Inexpensive Disks”, let me start by stating that was the original definition.

Over time, the definition has become more commonly known as “Redundant Array of Independent Disks” perhaps so the word “inexpensive” isn’t associated with RAID controllers or disks.

Personally I use the original definition but regardless, either definition means that the disks are independent of one another.

Feel free to use either definition since it won’t change the content of this article.

Now, back to our discussion of RAID.
When the original paper was issued, five different RAID levels or configurations were defined. Since that time other RAID configurations have been developed including what are referred to as “hybrid” RAID configurations.

The RAID Advisory Board (RAB) was created to help advise the IT community on the defined RAID configurations and to help the creation of new RAID configuration definitions.

While it is not an organization that creates legally binding standards and labeling, it does help in clarifying what the RAID levels mean and what is commonly accepted in the community.

There was a time where companies were creating very strange RAID configurations and using strange labels, causing great confusion.

The RAB has helped to reduce the proliferation of “weird” RAID configurations and labeling and standardize the meaning of various RAID levels.

In this article I want to review the seven most common standard RAID configurations. But I will also very briefly touch on some of the hybrid RAID configurations.

For each RAID level, I will describe how it works as well as the configuration’s particular pros and cons.

However, before starting I want to clarify one thing: RAID is not meant as a replacement for backups. RAID can help improve data reliability which really means data availability (improving uptime for data) and/or data performance (I/O performance).

It is not intended as a replacement for backups or keeping multiple independent copies of your data.

RAID Configurations

As mentioned above, there were five original RAID levels or configurations that were defined but others have been developed since that original article.

In RAID terminology each distinct RAID configuration is given a number which can also be called a RAID “level”.

The core RAID configurations are listed as: RAID-0, RAID-1, RAID-2, RAID-3, RAID-4, RAID-5, and RAID-6.


RAID-0
This RAID configuration is really focused on performance since the blocks are basically striped across multiple disks.

Figure 1 from wikipedia (image by Cburnett) illustrates how the data is written to two disks.

Figure 1: RAID-0 layout (from Cburnett at wikipedia under the GFDL license)

In this illustration, the first block of data, A0, is written to the fist disk, the second block of data, A1, is written to the second disk, the third block of data, A3, is written to the first disk, and so on.

If the I/O is happening fast enough data blocks can be written almost simultaneously (i.e. A0 and A1 are written at just about the same time).

Since the data is broken up into block sized units between the disks, it is commonly said that the data is striped across the disks.

As you can see, striping data across the disks means that the overall write performance of the disk set is very fast, usually much faster than a single disk.


Reading from a RAID-0 group is also very fast. A read request comes in and the RAID controller, which controls the placement of data, knows that it can read A0 and A1 at the same time since they are on separate disks, basically doubling the potential read performance relative to a single disk.


You can have as many disks as you want in a RAID-0 array (a group of disks in a RAID-0 configuration).

However, one of the downsides to RAID-0 is that there is is no additional data redundancy provided by RAID-0 (it is all focused on performance).

No data parity is computed and stored meaning that if you lose a disk in a RAID-0 array, you will lose access to all of the data in the array.

If you can bring the lost disk back into the array without losing any data on it, then you can recover the RAID-0 array, but this is a fairly rare occurrence.


Consequently, we can see that RAID-0 is focused solely on performance with no additional data redundancy beyond the redundancy in a single disk.

This affects how RAID-0 is used. For example, it can be used in situations where performance is paramount and you have a copy of your data elsewhere or the data is not important.

A classic usage case is for scratch space where data is written while an application is running but is not needed once the application is done and the final output is copied to a more resilient storage device.

If a scratch space disk is lost while the application is running, you can rebuild the RAID-0 array with one fewer drives, and rerun the application.


The capacity and failure rate of a RAID-0 array is the fairly simple to compute. The capacity is computed as,

Capacity = n * min(disk sizes)

where n is the number of disks in the array and min(disk sizes) is the minimum common capacity across the drives (this indicates that you can use drives of different sizes).

This equation also means that RAID-0 is very capacity effective since it doesn’t waste any space for parity or any other error correction. It uses all of the space for data focusing on performance.

The failure rate is a little more involved but can also be estimated.

MTTFgroup = MTTFdisk / n

where MTTF is the Mean Time To Failure and “group” refers to the RAID-0 array and “disk” refers to a single disk.

So as you add disks, you greatly reduce the MTTF for the RAID-0 array. Having two disks decreases the MTTF by half.

Three disks reduces the MTTF by a factor of 3, and so on. So you can tell why people are reluctant to use RAID-0 for file systems where data availability and reliability is important.

But, RAID-0 is the fastest RAID configuration and has the best capacity utilization of any RAID configuration discussed in this article.

Table 1 below is a quick summary of RAID-0 with a few highlights.


Table 1 - RAID-0 Highlights

Raid Level	Pros	Cons	Storage Efficiency	Minimum Number of disks
RAID-0	Performance (great read and write performance) Great capacity utilization (the best of any standard RAID configurations)	No data redundancy Poor MTTF	100% assuming the drives are the same size	2

RAID-1
RAID-1 is almost the exact opposite of RAID-0 because it uses multiple drives that are mirrors of one another.

Typically two drives are used in RAID-0 but three drive RAID-1 configurations are becoming more common.

RAID-1 takes an incoming block of data to one drive and creates a mirror image (copy) of it on a second drive.

So RAID-1 doesn’t compute any parity of the block - it just copies the entire block to a second drive.

Figure 2 from wikipedia (image by Cburnett) illustrates how the data is written to two disks in RAID-1.

Figure 2: RAID-1 layout (from Cburnett at wikipedia under the GFDL license)

In this illustration when block A1 is written to disk 0, the same block is also written to disk 1. Since the disks are independent of one another, the write to disk 0 and the write to disk 1 can happen at the same time.

However, when the data is read, the RAID controller can read block A1 from disk 0 and block A2 from disk 1 at the same time since the disks are independent.

So overall, the write performance of a RAID-1 array is the same as a single disk, and the read performance is actually faster from a RAID-1 array relative to a single disk.


The strength of RAID-1 lies in the fact that disks contains copies of the data. So if you lose disk 0, the exact same data is also on disk 1.

This greatly improves data reliability or availability.

The capacity of RAID-1 is the following:

Capacity = min(disk sizes)

meaning that the capacity of RAID-1 is limited by the smallest disk (you can use different size drives in RAID-1).

For example, if you have a 500GB disk and a 400GB disk, then the maximum capacity would be 400GB (i.e. 400GB of the 500GB drive is used as a mirror, and the remaining 100GB is not used).

RAID-1 has the lowest capacity utilization of any RAID configuration.

The reliability or probability of failure is also described in wikipedia. Since the disks are mirrors of one another but still independent, the probability of having both disks fail, leading to data lose, 

is the following:

P(dual failure) = P(single drive)2

So the probability of failure of a RAID-1 configuration is the square of the failure probability of a single drive.

Since the probability of failure of a single drive is less than 1, that means that the failure of a RAID-1 array is even smaller than the probability of failure of a single drive.


The reference has a more extensive discussion about the probability of failure but in general, the probably is fairly low.


One might be tempted to use RAID-1 for storing important data in place of backups of the data.

While RAID-1 improves data reliability or availability, it does not replace backups. If the RAID controller fails, or if the unit containing the RAID-1 array suffers some sort of failure, then the data is not available and may even be lost.

Without a backup you don’t have a copy of your data anymore. However, if you make a backup of the data, you would have a copy. The moral of the tale is - make real backups and don’t rely on 

RAID-1.

Table 2 below is a quick summary of RAID-1 with a few highlights.


Table 2 - RAID-1 Highlights

Raid Level	Pros	Cons	Storage Efficiency	Minimum Number of disks
RAID-1	Great data redundancy/availability Great MTTF	Worst capacity utilization of single RAID levels Good read performance, limited write performance	50% assuming two drives of the same size	2

RAID-2
This RAID level was one of the original five defined, but it is no longer really used. The basic concept is that RAID-2 stripes data at the bit level instead of the block level (remember that RAID-0 stripes at the block level) and uses a Hamming Coding for parity computations.

In RAID-2, the first bit is written on the first drive, the second bit is written on the second drive, and so on. Then a Hamming-code parity is computed and either stored on the disks or on a separate disk.

With this approach you can get very high data throughput rates since the data is striped across several drives, but you also lose a little performance because you have to compute the parity and store it.


A cool feature of RAID-2 is that it can compute single bit errors and recover from them. This prevents data errors or what some people call “bit rot”.

For an overall evaluation of RAID-2, there is this link.


According to this article hard drives added error correction that used Hamming codes, so using them at the RAID level became redundant so people stopped using RAID-2.


RAID-3
RAID-3 uses data striping at the byte level and also adds parity computations and stores them on a dedicated parity disk.

Figure 3 from wikipedia (image by Cburnett) illustrates how the data is written to four disks in RAID-3.

Figure 3: RAID-3 layout (from Cburnett at wikipedia under the GFDL license)

This RAID-3 layout uses 4 disks and stripes data across three of them and uses the fourth disk for storing parity information.

So a chunk of data “A” has byte A1 written to disk 0, byte A2 is written to disk 1, and byte A3 written to disk 3.

Then the parity of bytes A1, A2, and A3 is computed (this is labeled as A_p(1-3) in Figure 3) and written to disk 3.

The process then repeats until the entire chunk of data “A” is written. Notice that the minimum number of disks you can have in RAID-3 is three (you need 2 data disks and a third disk to store the parity).

RAID-3 is also capable of very high performance while the addition of parity gives back some data reliability and availability compared to a pure striping model ala’ RAID-0.

Since the number of disks in a stripe is likely to be smaller than a block all of the disks in a byte-level stripe are accessed at the same time improving read and write performance.

However, the RAID-3 configuration some possible side effects.


In particular, this link explains that RAID-3 cannot accommodate multiple requests at the same time.

This results from the fact that a block of data will be spread across all members of the RAID-3 group (minus the parity disk) and the data has to reside in the same location on each drive.

This means that the disks (spindles) have to be accessed at the same time, using the same stripe, which usually means that the spindles have to be synchronized.

As a consequence, if an I/O request for data chunk A comes into the array (see Figure 3), all of the disks have to seek to the beginning of the chunk A and read their specific bytes and send it back to the RAID-3 controller.

Any other data request, such as that for a data chunk labeled B in Figure 3 is blocked until the request for “A” has completed because all of the drives are being used.

The capacity of RAID-3 is the following:

Capacity = min(disk sizes) * (n-1)

meaning that the capacity of RAID-3 is limited by the smallest disk (you can use different size drives in RAID-3) multiplied by the number of drives n, minus one.

The “minus one” part is because of the dedicated parity drive.


RAID-3 has some good performance since it is similar to RAID-0 (striping), but you have to assume some reduction in performance because of the parity computations (this is done by the RAID controller).

However, if you lose the parity disk you will not lose data (the data remains on the other disks). If you lose a data disk, you still have the parity disk so you can recover data.

So RAID-3 offers more data availability and reliability than RAID-0 but with some reduction in performance because of the parity computations and I/O.

More discussion about the performance of RAID-3 is contained at this link.


RAID-3 isn’t very popular in the real-world but from time to time you do see it used.

RAID-3 is used in situations where RAID-0 is totally unacceptable because there is not enough data redundancy and the data throughput reduction due to the data parity computations is acceptable.

Table 3 below is a quick summary of RAID-3 with a few highlights.


Table 3 - RAID-3 Highlights

Raid Level	Pros	Cons	Storage Efficiency	Minimum Number of disks
RAID-3	Good data redundancy/availability (can tolerate the lose of 1 drive) Good read performance since all of the drives are read at the same time Reasonable write performance but parity computations cause some reduction in performance Can lose one drive without losing data	Spindles have to be synchronized Data access can be blocked because all drives are accessed at the same time for read or write	(n - 1) / n where n is the number of drives	3 (have to be identical)

RAID-4

RAID-3 improved data redundancy by adding a parity disk to add some reliability. In a similar fashion, RAID-4 builds on RAID-0 by adding a parity disk to block-level striping.

Since the striping is now down to a block level, each disk can be accessed independently to read or write data allowing multiple data access to happen at the same time.

Figure 4 below from wikipedia (image by Cburnett) illustrates how the data is written to four disks in RAID-4.

Figure 4: RAID-4 layout (from Cburnett at Wikipedia under the GFDL license)

In this layout, data is written in block stripes to the first three disks (disks 0, 1, and 2) while the third drive (disk 3) is the parity drive.

The parity of the blocks across the drives is computed by the RAID controller and stored on the dedicated parity drive.

In the figure The parity for A1, A2, and A3 is listed as A_p on the parity drive.


The dedicated parity drive becomes a performance bottleneck in RAID-4, particularly for write I/O.

Since RAID-4 has block level striping, you can write to blocks A1 and B2 at the same time since they are on different disks.

However, the parity for both blocks has to be written to the same drive which can only accommodate a single write I/O request at a time.

Consequently, one of the parity writes (A1 parity or B2 parity) is blocked and the write I/O performance is reduced.

For more on the performance of RAID-4, please see this link.

The capacity of RAID-4 is the following:

Capacity = min(disk sizes) * (n-1)

meaning that the capacity of RAID-4 is limited by the smallest disk (you can use different size drives in RAID-4) multiplied by the number of drives n, minus one.

The “minus one” part is because of the dedicated parity drive.

However, it is recommended you use drives that are the same size in RAID-4.


RAID-4 improves on the redundancy of RAID-0, which has zero data redundancy, by adding a parity disk.

You can lose one drive without losing data. For example you could lose the parity disk without losing data or you could lose one of the data disks without losing data.

But the introduction of the single dedicated parity drive has reduced write performance relative to RAID-0.

However, if the loss of write performance of RAID-4 is acceptable it does give you more data redundancy than RAID-0.


RAID-4 was the last RAID configuration defined in the original RAID paper.

In the real-world, RAID-4 is rarely used because RAID-5 (see next sub-section) has replaced it.

Table 4 below is a quick summary of RAID-4 with a few highlights.

Table 4 - RAID-4 Highlights

Raid Level	Pros	Cons	Storage Efficiency	Minimum Number of disks
RAID-4	Good data redundancy/availability (can tolerate the lose of 1 drive) Good read performance since all of the drives are read at the same time Can lose one drive without losing data	Single parity disk (causes bottleneck) Write performance is not that good because of the bottleneck of the parity drive	(n - 1) / n where n is the number of drives	3 (have to be identical)

RAID-5
RAID-5 is similar to RAID-4 but now the parity is distributed across all of the drives instead of using a dedicated parity drive.

This greatly improves write performance relative to RAID-4 since the parity is written on all of the drives in the RAID-5 array.

Figure 5 below from wikipedia (image by Cburnett) illustrates how the data is written to four disks in RAID-5.

Figure 5: RAID-5 layout (from Cburnett at wikipedia under the GFDL license)

In this layout, the parity blocks are labeled with a subscript “p” to indicate parity. Notice how they are distributed across all four drives.

The blocks that line up (one block per drive) are typically a “stripe”. In Figure 5 the blocks in a stripe are all the same color.

The data stripe size is simply the following:

Data stripe size = block size * (n-1)

where n is the number of drives in the RAID-5 array. Inside a stripe there is a single parity block and all other blocks are data blocks.

Anytime a block inside the stripe is changed or written to, the parity block is recomputed and rewritten (this is sometimes called the read-modify-write process).

This process can add overhead reducing performance.


RAID-5 also has some write performance problems for small writes that are smaller than a single stripe since the parity needs to be computed several times which eats up computational capability of the RAID controller.

As mentioned previously the read-modify-write process that must be followed happens much more often in this case.

The capacity of RAID-5 is very similar to RAID-4 and is the following:

Capacity = min(disk sizes) * (n-1)

meaning that the capacity of RAID-5 is limited by the smallest disk (you can use different size drives in RAID-5) multiplied by the number of drives n, minus one.

The “minus one” part is because of the parity block per stripe.


With RAID-5 you can lose a single drive and not lose data because either the data or the parity for the missing blocks on the lost drive can be found on the remaining drives.

In addition, many RAID controllers allow what is called a hot-spare drive. This drive is typically part of the RAID array but is initially not used for storing data.

If the RAID group loses a drive, the hot-spare is immediately brought into the RAID group by the controller.


In the case of RAID-5, the controller immediately starts redistributing data and parity blocks to this new drive.

To do this, the initial drives in the RAID-5 array have to have all blocks read and the RAID controller has to recompute parity or rebuild missing data blocks.

This combination means that it can take quite a bit of time to fail-over data to the hot-spare drive.

The nice thing about having a hot-spare drives is that typically the fail-over process happens automatically so there is almost no delay in incorporating the hot-spare drive.


RAID-5 has been used for a very long time and during this time the data availability and redundancy has been very good.

However, there is a new phenomenon that impacts RAID-5 that has been explained in various article around the web such as this one.

Basically the capacity of drives is growing quicker than the Unrecoverable Read Error (URE) rate of drives to the point where losing a drive in a RAID-5 array and recovering it to a hot-spare drive is almost guaranteed to lead to a URE which means that the RAID-5 array will be lost and the data has to be restored from a backup.

However, this is the subject for another article.


There is no shortage of articles about RAID-5 on the web. You will see some strong opinions both for and against RAID-5 based on usage cases.

Be sure to understand the application used when reading about both pros and cons of RAID-5.

A reasonable overview of the trade-offs of RAID-5 is this article.

Table 5 below is a quick summary of RAID-5 with a few highlights.


Table 5 - RAID-5 Highlights

Raid Level	Pros	Cons	Storage Efficiency	Minimum Number of disks
RAID-5	Good data redundancy/availability (can tolerate the lose of 1 drive) Very good read performance since all of the drives can be read at the same time Write performance is adequate (better than RAID-4) Can lose one drive without losing data	Write performance is adequate (better than RAID-4) Write performance for small I/O is not good at all	(n - 1) / n where n is the number of drives	3 (have to be identical)

RAID-6
As mentioned previously, there is a potential problem with RAID-5 for larger capacity drives and a larger number of them.

RAID-6 attempts to help that situation by using two parity blocks per stripe instead of RAID-5’s single parity block.

This allows you to lose two drives with losing any data. Figure 6 below from wikipedia (image by Cburnett) illustrates how the data is written to four disks in RAID-6.

Figure 6: RAID-6 layout (from Cburnett at wikipedia under the GFDL license)

In this figure, the first parity block is noted with as subscript “p” such as A_p. The second parity block in a stripe is noted with a subscript “q” such as A_q.

The use of two parity blocks reduces the useable capacity of a RAID-6 as in the following:

Capacity = min(disk sizes) * (n-2)

meaning that the capacity of RAID-6 is limited by the smallest disk (you can use different size drives in RAID-6) multiplied by the number of drives n, minus two.

The “minus two” part is because of the two parity blocks per stripe.


Computing the first parity block, p, is done in the same fashion as RAID-5. However, computing the q parity block is more complicated as explained here.

This means that the write performance of a RAID-6 array can be slower than a RAID-5 array for a given level of RAID controller performance.

However, read performance from a RAID-6 is just as fast as a RAID-5 array since reading the parity blocks is skipped. But in exchange for worse performance, RAID-6 arrays can tolerate the lose of two drives while RAID-5 can only tolerate the lose of a single drive.

Coupled with larger drives and larger drive counts, this means that larger RAID-6 arrays can be constructed realtive to RAID-5 arrays.

Table 6 below is a quick summary of RAID-6 with a few highlights.


Table 6 - RAID-6 Highlights

Raid Level	Pros	Cons	Storage Efficiency	Minimum Number of disks
RAID-6	Excellent data redundancy/availability (can tolerate the lose of 2 drives) Very good read performance since all of the drives can be read at the same time Can lose two drives without losing data	Write performance is not that good - worse than RAID-5 Write performance for small I/O is not good at all more computational horsepower is required for parity computations	(n - 2) / n where n is the number of drives	4 (have to be identical)

Hybrid RAID Levels

As you can see, there are some limitations to each of the standard RAID levels (0-6). Some of the them have great performance (RAID-0) but pretty awful data availability or redundancy while others have very good data availability and redundancy (RAID-6) but the performance is not so hot.

So as you can imagine, people started to wonder if they couldn’t combine RAID levels to combine features to perhaps achieve better performance while still having very good data redundancy and availability.

This lead to what people called Hybrid RAID Levels or what is more commonly called Nested RAID levels.



The topic of Nested RAID levels is fairly lengthy so I will save that for another article. But the basic concept is to combine RAID levels in some fashion.

For example, a common configuration is called RAID 1+0 or RAID-10. The first number (the furthest to the left) refers to the “bottom” or initial part of the RAID array.

Then the second number from the left refers to the “top” level or the RAID array. The top level RAID uses the bottom level RAID configurations as building blocks.


In the case of RAID-10, the approach is to use multiple pairs of drives at the lowest level (RAID-1) and then to combine them using RIAD-0.

This retains the goodness of RAID-1 for data availability and redundancy while gaining back some performance from RAID-0 striping.

Summary

This wraps our introduction to RAID. For some people it may be new and for many it will be review.

Now that we’ve covered the basics, in coming articles we will be exploring Nested-RAID more in depth, including RAID-01 RAID-5, RAID-6 and RAID-10 configurations.

Have questions about RAID or topics you’d liked to see covered? Post them in the comments and we’ll try to incorporate them as deep dive into redundant arrays. 

4 Open Source Applications for the Visually Impaired

When you're visually impaired, getting around the Internet isn't always easy. Fortunately, the open source community takes accessibility very seriously.

If you or someone you know needs assistive computer software, here are four open source applications you'll want to check out.

Qwitter - Orignally designed as a Twitter client for the visually impaired, Qwitter has become a more fully-formed application that works across the internet with most screen readers, including the open source project NVDA for Windows.

Use it to search Bing, interact with Captcha-completer service Solona, manipulate your Google Voice account, and more.

Orca - This screen reader is bundled with the GNOME desktop (version 2.16 and newer), and ships with the Open Solaris, Ubuntu, and Fedora operating systems, but it is also available for separate download.

It supports the OpenOffice word processing suite and Firefox browser, and the Java platform, making Orca one of the most versatile open source screen readers available.

It also works with an ever-growing assortment of stand-alone apps.

WebAnywhere - Unlike screen readers that must be installed on a PC in order to work correctly, this Web-based tool requires no installation.

WebAnywhere works on any computer with a sound card, so visually impaired people can use publicly accessible PCs (at the airport or library, for instance) or borrow a friend's laptop hassle-free.

The vOICe for Android - Designed specifically for Android phones, this app uses sonic augmented reality with a device's built-in camera to provide geo-spatial feedback to the user via sound.

Check the app's website for full details on how it works, or download it for free in the Android Market by searching "seeingwithsound."

Checking out JQuery Mobile

JavaScript

In a seemingly short period of time, JavaScript has gone from a form-validation tool to a game changer.

Having my head buried into SDK development for a number of years, I have taken relatively small notice of JavaScript until this past 18 months.

It’s no longer just a scripting language for pulling off Web2.0 tricks but a mainstream language for building full-blown applications.

In the mobile world, one example of this is WebOS, which came onto the scene with an innovative and flexible user interface, driven by JavaScript, HTML and CSS.

While WebOS has a long way to go to prove that it is more than just a “drive-by operating system”, one thing is for sure, JavaScript is here for a good while.

And WebOS isn’t the only driver — there are other champions of JavaScript to note.

PhoneGap has become a player in the mobile app development scene as developers are leveraging their web skills to build mobile applications with an open source, royalty free framework.

And the folks at Appcelerator have some non-trivial venture funds behind them and have gone through a number of iterations all built around the JavaScript/HTML/CSS model.

While their approach has shifted a bit over time, they are committed to the rapid development capabilities that JavaScript can enable.

So, it would seem that web technologies have a prominent place in mobile as WebOS, PhoneGap and Appcelerator have each taken their own approach to bending web technologies to their mobile will.

But what about true web applications — where do they fit in?

While there are some techniques available for making a “normal” web page mobile friendly — things like setting view-port and scaling the page appropriately to make sure that the page renders nicely on a mobile device, the truth is that the WebKit browser engine, now standard in many major mobile platforms, renders content quite nicely, mobile-optimized or not.

So, if web content renders nicely on a mobile browser, do we care about building web apps “for mobile” at all?

Though the question alone is worthy of debate over a beverage of your choice, it is sometimes helpful to have a look at the market itself to find our answer. And we need not look any further than the folks at JQuery.com.

A Familiar Player

The JQuery project has brought advanced web functionality to the masses and they are now getting into mobile in a big way.

In fact, the impact of this mobile effort might make some think that the tail is wagging the dog here as the JQuery team is introducing JQuery Mobile.

The JQuery mobile effort is not only introducing new capabilities for mobile developers but the effort is causing the JQuery team to go back to their core library and revisit architectural and performance characteristics.

All of this is good news for developers, web users, and now mobile users also.

The JQuery Mobile project encompasses some major functionality and is currently at the Alpha 2 Release.

The balance of this article takes a look at some of the goals of this project and in upcoming articles we will demonstrate some of JQuery’s major features.

The basics

The JQuery team is targeting all major mobile browser options. Patterned after Yahoo’s browser grading system, JQuery Mobile aims to offer their complete array of functionality on “A” level browsers.

Most of the platforms in the list have native browsers capable of advanced functionality, however in some cases the better option is currently delivered by the Opera browser.

In case you are wondering just how many “major” mobile platforms exist today, here is the list of the primary targets world-wide:

• iPhone/iPod/iPad
• Symbian Series 60
• Symbian UIQ
• Symbian Platform (3.0)
• BlackBerry OS (primarily Torch and later)
• Android
• Windows Mobile (sort of)
• webOS
• bada
• Maemo
• MeeGo

A primary objective of JQuery Mobile is to create a library where truly cross-platform applications may be written, eliminating (OK, reducing) the need for platform-specific applications.

If they can deliver on this it will lower the barrier to entry for many software vendors struggling to bring their products to the mobile scene.

A viable UI framework based on web tools enables virtually every software vendor to access the mobile space.

Another winner in this trend is the tablet class of device as tablets look to play an increasingly important role in both the consumer and commercial spaces.

The availability of a clean, consistent web-based application presentation framework will only accelerate the penetration tablet devices enjoy.

So, what all does JQuery include? Lots. Here is a brief list of some features of JQuery mobile.

• Built on the core JQuery library
• Quick runtime download via a small footprint, similar to JQuery core
• Utilizes HTML5 based markup for simplifying the development process
• Relies upon progressive enhancement to deliver high quality features on capable browsers and degrade acceptably on lower-end browsers
• Built with screen readers and other accessibility features in mind from the outset
• Accepts diverse input devices — touch, mouse and cursor input methods
• Upgrades HTML Form elements with new, snazzy controls
• Customizable with powerful theming framework

One of the nicest features of developing mobile applications with JQuery Mobile is that you don’t need to do 100% of your testing on the small screen of a phone.

Because the mobile browsers are often built upon WebKit, developers can use Safari, the WebKit nightly build, or Chrome to proto-type applications. Of course, don’t ignore the mobile browser altogether!

I am increasingly finding myself testing web pages in Safari due to this high level of compatibility.

Speaking of testing, in our next article we will build a basic application, exercising a multi-page application, highlighting some of the UI elements included in the JQuery Mobile library.

Until then, you can read more about JQuery Mobile directly at http://jquerymobile.com.

Car Theft by Antenna

Researchers beat automatic locking and ignition systems.

Car thieves of the future might be able to get into a car and drive away without forced entry and without needing a physical key, according to new research that will be presented at the Network and Distributed System Security Symposium next month in San Diego, California.

The researchers successfully attacked eight car manufacturers' passive keyless entry and start systems—wireless key fobs that open a car's doors and start the engine by proximity alone.

Srdjan Capkun, an assistant professor of computer science in the system security group at ETH Zurich in Switzerland, who led the work, says he was inspired to investigate the security of keyless entry and start systems after buying a car that had one.

Capkun and Aurélien Francillon and Boris Danev, both researchers in the same institution, examined 10 car models from the eight manufacturers.

They were able to access all 10 and drive them away by intercepting and relaying signals from the cars to their wireless keys.

While they could relay the signals from the key back to the car as well, usually they did not need to because the key transmits its signals up to around 100 meters.

The attack works no matter what cryptography and protocols the key and car use to communicate with each other.

Normally, when a wireless key is within a few meters of the right car, it detects a low-powered signal that causes it to issue a command that opens the car enable the ignition.

The researchers used a pair of antennas to transmit these signals from the car to the key when the key was farther away, tricking the car into opening without the ordinary authorization.

One antenna needs to be very close to the car, and one needs to be within eight meters of the key.

The researchers came up with two versions of the attack. In one, they ran a cable from near the car to near the key and used it to transmit the signals.

They conducted the other wirelessly. Francillon says that the materials for the wired attack cost about $50, and those for the wireless attack cost between $100 and $1,000, depending on the electronic components used.

The researchers tested a few scenarios. An attacker could watch a parking lot and have an accomplice watch as car owners as entered a nearby store.

The accomplice would only need to be within eight meters of the targeted owner's key fob, making it easy to avoid arousing suspicion.

In another scenario, a car owner might leave a car key on a table near a window. An antenna placed outside the house was able to communicate with the key, allowing the researchers then to start the car parked out front and drive away.


A car won't open or start if the signal from its key takes too long to arrive, so the researchers devised a way to speed communication between their antennas.

Most relay attacks require the signals to be converted from analog to digital and back, which takes time.

The researchers were able to keep the signals in analog format, which reduced their delay from microseconds to nanoseconds and made their attack more difficult to detect.

The researchers suggest things that car owners and manufacturers can do to protect themselves. Car owners can shield their keys when they're not in use, to prevent attackers from communicating with them.

Alternatively, manufacturers could add a button to fobs that would allow owners to deactivate and reactivate them.

Capkun worries, however, that these types of solutions detract from the convenience that makes passive keyless entry systems worthwhile.

Ultimately, he says, manufacturers will need to add secure technology that allows the car to confirm that the key is in fact nearby. "I don't see a way around it," Capkun says. His group is actively working on protocols that would accomplish this.

David Wagner, a professor of computer science at the University of California at Berkeley who has studied the cryptographic systems used in keyless entry systems, says the research "should help car manufacturers improve auto security systems in the future."

Wagner doesn't think the research ought to make car owners anxious. "There are probably easier ways to steal cars," he says. But, he adds, a "nasty aspect of high-tech car theft" is that "it doesn't leave any sign of forced entry," so if a thief did use this method to steal a car, he says, it might be hard for police and insurance companies to get sufficient evidence of what happened. Wagner believes that manufacturers, police, and insurance companies all need to prepare for this eventuality.

"Automobiles are a key example of a system that is pervasively computerized," so they need to be thoroughly examined to ensure they are secure, says Tadayoshi Kohno, an assistant professor of computer science at the University of Washington.

Kohno helped form the Center for Automotive Embedded Systems Security, which is dedicated to identifying and solving security problems with car security systems before they cause problems in the real.

Hackers find new way to cheat on Wall Street -- to everyone's peril

Regarding this article I am not posting it as a security incident or news at all. It contains wealth of information about bandwidth and throughput and how we perceive them. It also illustrates how network monitoring and measurement should take place.
Enjoy!!!
--------------------------------------------------------------------------------------------------------------------------

Side-channel attack on high-frequency trading networks could net a hacker millions of dollars in seconds -- and leave everyone else much poorer

High-frequency trading networks, which complete stock market transactions in microseconds, are vulnerable to manipulation by hackers who can inject tiny amounts of latency into them. By doing so, they can subtly change the course of trading and pocket profits of millions of dollars in just a few seconds, says Rony Kay, a former IBM research fellow and founder of cPacket Networks, a Silicon Valley firm that develops chips and technologies for network monitoring and traffic analysis.

Kay, an Israeli-born computer scientist and one-time Intel engineering manager, says the root of the problem is the increasing speed of networks; as they get faster and faster, our ability to actually understand events taking place within them isn't keeping up. Network monitoring technology can detect perturbations in network traffic happening in milliseconds, but when changes occur in microseconds, they're not visible, he says.

[ For the key tech news of the day, sign up for InfoWorld's Tech Headlines Wrap-Up newsletter. | Learn how to greatly reduce the threat of malicious attacks with InfoWorld's Insider Threat Deep Dive PDF special report. ]

cPacket has developed a proof of concept showing that these side-channel attacks can be used to create tiny delays in the transmission of market data and trades. By manipulating specific trading activities by several microseconds, an attacker could gain unfair trading advantage. And because the operation occurs outside the range of monitoring technology, it would remain invisible. "We believe that such techniques pose a substantial risk of creating unfair trading, if used by the wrong people," Kay says.

(A side-channel attacker looks at indirect information related to the computer -- the electromagnetic emanations from screens or keyboards, for example -- to determine what is going on in the machine. )

Latency threatens other applications as well
The lack of visibility into high-speed networks is of concern to more than the financial community. Managing traffic on today's 10Gbps and faster networks is becoming difficult, resulting in degradations of performance, particularly to virtualized systems. "It's difficult to take corrective actions when you can't really see what's taking place," Kay says. "If you cannot measure network latency, you cannot control it and cannot improve it."

In a PDF whitepaper on latency, Kay wrote, "Traditionally, applications that have latency requirements include: VoIP and interactive video conferencing, network gaming, high-performance computing, cloud computing, and automatic algorithmic trading. For example, one-way latency for VoIP telephony should generally not exceed 150 milliseconds (0.15 seconds) to enable good conversation quality, while interactive games typically require latencies between 100 and 1,000 milliseconds. However, the requirements for automated algorithmic trading are much more strict. A few extra milliseconds, or even a few extra microseconds, can enable trades to execute ahead of the competition, thereby increasing profits."

Indeed, latency, even at the very highest speeds, is so concerning that researchers at MIT recommended any organization dealing in complicated time-sensitive global interactions should take a hard look at where they locate their data centers.

The MIT researchers even suggested that financial firms could gain some advantage by taking advantage of limitations posed by the speed of light. For example, it typically takes about 50 milliseconds to send a message from New York to London. Placing a server between the two could cut the speed of communication in half, they said, which may be enough time to take advantage of some momentary pricing discrepancy. Trading on that discrepancy is known as arbitrage, and it's becoming increasingly common.

Lessons of the "flash crash"
The vulnerability of markets in which high-frequency trading is common became all too evident last May, when exchanges experienced a "flash crash" that drove the Dow Jones down about 600 points in just five minutes. The incident was not the result of deliberate manipulation, but it shows just how dependant the financial world is on technology it doesn't really understand.

"Financial institutions and exchanges with [high-frequency trading] are spending millions to improve latency by microseconds and at the same time can't measure the data at that resolution in real time. It's disturbing," Kay says.

A side-channel attack on a high-frequency trading network is analogous to a denial-of-service attack. In a typical DoS attack, bots flood a target website with enormous numbers of hits, often causing a crash. A side-channel attack would be infinitely more subtle, but it would still function by adding extraneous packets to a legitimate data stream. Those extra packets slow the data just enough to give someone else a chance to move first in the market.

Kay says he does not know if anyone has yet launched a side-channel attack against a high-frequency trading network -- but it worries him. And it worries me. Financial markets are supposed to be a level playing field. They're not, of course. Small players, like the millions of us who invest for our 401(k)s and other retirement accounts, are at an immense disadvantage even when everything is kosher. But the proliferation of high-frequency trading widens the gap even more. If someone can really take advantage of a weakness in those networks, we're all really in trouble. And that's just another reason why more -- not less -- regulation is required in the financial markets.

I welcome your comments, tips, and suggestions. Post them here so that all our readers can share them, or reach me at bill.snyder@sbcglobal.net. Follow me on Twitter at BSnyderSF.

This article, "Hackers find a new way to cheat on Wall Street -- to everyone's peril," was originally published by InfoWorld.com. Read more of Bill Snyder's Tech's Bottom Line blog and follow the latest technology business developments at InfoWorld.com.

How to secure your Linux system

Are you running Linux just because you think it's safer than Windows? Think again. Sure, security is a built-in (and not a bolt-on) feature and extends right from the Linux kernel to the desktop, but it still leaves enough room to let someone muck about with your /home folder.

Linux might be impervious to viruses and worms written for Windows, but that's just a small subset of the larger issue.

Attackers have various tricks up their sleeves to get to those precious bits and bytes that make up everything from your mugshot to your credit card details.

Computers that connect to the internet are the ones most exposed to attackers, although computers that never get to see online action are just as vulnerable.

Think of that ageing laptop or that old hard disk you just chucked away without a second thought. Bad move.

With the kind of data recovery tools available today (many as a free download) it doesn't matter what OS was installed on the disk.

If it holds data – corrupted or otherwise – it can be retrieved, bank accounts recreated, chat transcripts reconstructed, images restitched.

But don't be scared. Don't stop using the computer.

While it's virtually impossible to make a machine connected to the internet impenetrable to attacks, you can make an attacker's task difficult and also ensure they have nothing to learn from a compromised system.

Best of all, with Linux, and some pieces of open source software, it doesn't take much effort to secure your Linux installation.

There is no golden rule for security that applies in every single case, and even if there were it would have been cracked already.

Security is something that needs to be worked upon, and personalised. Follow the tips and tools in this tutorial as we show you how to adapt them to your very own Linux installation.

Follow these six tips to get a safer computer the easy way
1. Keep up with security updates


All mainstream Linux desktop distros (such as Debian, Ubuntu, Fedora, etc) have security teams that work with the package teams to make sure you stay on top of any security vulnerabilities.

Generally these teams work with each other to make sure that security patches are available as soon as a vulnerability is discovered.

Your distro will have a repository solely dedicated to security updates.

All you have to do is make sure the security specific repository is enabled (chances are it will be, by default), and choose whether you'd like to install the updates automatically or manually at the press of a button.

For example, under Ubuntu, head over to System > Administration > Software Sources. Here, under the Updates tab, specify how frequently the distro should ping the security repository for updates, and whether you'd like to install them without confirmation, or just be notified about the updates.

The latter is a better option, because it lets you review the updates before installing them. But chances are they'll be fine, and you can save yourself some time by having your distro install them automatically.

In addition to the updates, distros also have a security mailing list to announce vulnerabilities, and also share packages to fix them.

It's generally a good idea to keep an eye on the security list for your distro, and look out for any security updates to packages that are critical to you.

There's a small lag between the announcement and the package being pushed to the repository; the security mailing lists guide the impatient on how to grab and install the updates manually.


2. Disable unnecessary services


A Linux desktop distro starts a number of services to be of use to as many people as possible. But one really doesn't need all these services.

For example, do you really need Samba for sharing files over the network on your secure server, or the Bluetooth service to connect to Bluetooth devices on a computer that doesn't have a Bluetooth adapter?

All distros let you control the services that run on your Linux installation, and you should make full use of this customization feature.

Under Ubuntu, head to System > Preferences > Startup Applications. Here you can remove check marks next to the services you wish to disable.

But be careful when turning off services. Some applications might stop functioning because you decided to disable a service on which they rely.

For example, many server applications rely on databases, so before you turn off MySQL or PostgreSQL you should make sure you aren't running any applications that rely on them.


3. Restrict root access


Most distros these days don't allow you to login as root at boot time, which is good. When you have to execute a task that requires super user privileges you'll be prompted for a password.

It might be a little irritating but it goes a long way to making sure that admin tasks are isolated from the user.

You can restrict access privileges for a user from under System > Administration > Users and Groups.

Here you can broadly categorise a user as a desktop user or a system administrator or customise access privileges manually.

By default, users are created as with 'Desktop user' permissions and can't install software or change settings that affect other users.

On the command line, the su command (on Fedora, and the like) lets normal users switch to the root account, while the sudo command (on Debian, Ubuntu, etc) grants more privileges to the user.

The usage of these commands can be limited to a particular group, which prevents any user from administering the system. sudo is also the more secure of the two, and it keeps an access log under /var/log/auth.log.

Make a habit of regularly scanning the log for failed and successful sudo attempts.

4. Don't auto-mount devices


If you're really concerned about security, you need to lean on the customisation feature of the Users And Groups settings. One of the areas to look at is auto-mounting devices.

Most distros auto-mount USB drives and CDs as soon as they are inserted. It's convenient, but allows anybody to just walk up to your machine, plug in a USB disk and copy all your data.

To avoid such a situation, go to to System > Administration > Users and Groups, select your user and head to the Advanced Settings > User Privileges tab.

Make sure you uncheck the boxes corresponding to the Access External Storage Devices Automatically option, the Mount Userspace Filesystems, and Use CD-ROM Drives option.

When unchecked, these options will prompt the user for a password before giving them access to these devices.

You might also want to disable sharing files on the network, as well as require the user to enter a password before connecting to the Ethernet and wireless devices.

By disabling access to configure printers you prevent important data from being printed.

5. Don't stay on the bleeding edge


Packages included in a desktop Linux distribution are updated regularly. Besides the official repositories, there are custom repositories for third-party software.

While developers do take care to scan the packages for vulnerabilities before pushing them on to the repository, it's almost inevitable that some updates with defects do get through.

While it's good to keep the system updated, from a security point of view, not all updates are good for the system.

Some updates conflict with existing installed package or may even pull in new dependencies that may make the system more prone to attack. All this is why you should only update packages if you have to.

Scan the updates and look for updates to packages that are critical to you. Most package managers also make it possible to check an update and display its changelog and a brief description of the changes.

UI changes can safely be ignored or delayed until a package has been thoroughly tested. Instead, look out for and grab updates that offer a fix to existing issues with packages.

6. Don't upgrade every six months


Most major desktop Linux distributions make a new release every six months, but you don't have to install every last upgrade just because it's there.

Debian, for example, offers three distributions to choose from based on the extent of the stability of the software available in it. After Debian 6.0, stable releases will be made every two years.

Other distros take a different approach to guarantee secure releases. Ubuntu marks certain releases as LTS (or Long Term Support).

A desktop release of the LTS version is supported for three years, and a server release is supported for five years, which is a lot longer than the 18 months for a standard Ubuntu release.

Although not up to date, these releases are much more secure from a security point of view, with packages that are a lot more stable and more thoroughly tested than their latest versions.

If running a secure system is your goal, you should think of sticking to one of these long-term stable releases and avoid the temptation to upgrade as soon as the latest version of your becomes available.

 Out of the box, a Linux installation is much more secure than other operating systems. That is, until you connect to the internet. Once online, a desktop Linux installation, in its bid to be of use to as many users as possible, leaves enough room to be exposed to attacks and intrusions.

Don't sweat though. Help is only a terminal away.

All Linux distros ship with Iptables, which is a part of the kernel that enables sysadmins to filter network packets.

Configuring it manually is impossible for all but the elite, but in the true spirit of open source the community offers a number of graphical front-ends that make setting up a firewall a walk in the park. One such graphical firewall is Firestarter.




We didn't start the fire
Firestarter simplifies the process of configuring the settings for a firewall. It can limit access on ports that are running services that might be prone to outside attacks, and you can also use it to glance at the network traffic passing across the machine you're running it on.

Most distros bundle Firestarter in their repos, so installing it shouldn't be a problem. When you start it for the first time, the firewall launches a simple configuration wizard that prompts you to select the network interface on which it will be active.

If you have multiple devices with one connecting to the internal network, Firestarter can act as gateway and share the internet connection with the rest of the network.

By default, Firestarter only filters through connections that are in response to connection requests from the firewall host.

The advantage of doing things this way is that it blocks access to services like Telnet, which can be exploited to gain access to your machine without your knowledge.

Tweaking the firewall doesn't take much effort either. If you have an app that requires access on certain ports, such as a Torrent client, you need to punch holes in your firewall to allow incoming connections. That's easily done from under the Policy tab.

Right-click inside the space under Allow Service and select Add Rule. From the pull-down menu, select the service you want to allow, say Samba, select the source IP (anyone opens the port to all) and you're done.

To restrict outgoing traffic, select Outbound Traffic Policy from the drop-down list. Now you can select either the Permissive or the Restrictive option.

If you select the Permissive option, you'll have to add the hosts you want to block in a blacklist.

Restrictive is the opposite, and only allows connections from the listed hosts, denying the rest.

When running in restrictive mode, Firestarter will log all connection refusals under the Events tab. As you spot a connection you want to allow for your users, right-click on the entry and select the option to either allow the connection for everyone or just when it originates from a particular source.

You can also monitor active connections to the firewall from Firestarter's main interface. It shows you the status of the service, gives you a summary of inbound and outbound connections, and the amount of data that has passed through an interface.

In addition to listing the source and destination of the traffic, it'll also tell you the port the data is travelling through, the service running on that port and the program that's calling the shots.


Encrypt your filesystem
If you really want to keep others from reading your files, user passwords won't cut it. For instance, there's very little to stop a user with higher access permissions, like the root user, from gawking at stuff under your home directory.

What you need is to encrypt your data so that it's unintelligible to people without the means to decrypt it.

The smart way to do this is to encrypt the whole filesystem, which would automatically encrypt any data kept on it. This is where TrueCrypt shines.



It lets you carve a virtual slice out of your Linux partition that will act as a standalone encrypted filesystem.

You then mount it, use it to store and read files as you would from a normal partition, then unmount it, and Bob's your uncle.

When it isn't mounted, the encrypted filesystem appears to be a random jumble of bits.

TrueCrypt isn't available in any distribution's repository due to licensing issues, but installing it is a trivial affair.

Grab it from its website, extract the Tar archive, and install it via the graphical setup. Just make sure your distro has the Fuse library, and the device mapper tools.


Create an encrypted volume
Before you can use TrueCrypt you'll have to create an encrypted volume to store files on, so launch the app and click on the Create Volume button.

This will launch the Volume Creation Wizard, which lets you either create a virtual encrypted disk within a file or an encrypted volume within an entire partition, or even a disk such as a removable USB drive.

If you select the first option to create a virtual disk, TrueCrypt will ask you to point it to a file on the disk that'll be the encrypted volume.

If the file exists, TrueCrypt will recreate it, using one of the eight encryption algorithms.

Next, specify the size of the encrypted volume and format it as an FAT filesystem, which makes it accessible from other operating systems as well as Linux.

Finally, choose a password to mount the encrypted volume.

To store files on the volume you'll have to mount it. Select the file that's your encrypted volume from the TrueCrypt main interface, and press the Mount button.

The app will prompt for the password of the volume before it can be mounted. You also get the option to mount the volume as read-only, if all you have to do is read files from it.

By default, TrueCrypt chooses not to remember the name of the file that's your encrypted volume. This is a security feature, and adds another roadblock in the path of an intruder.

If you ask the app to remember the name of the file, anyone with physical access to the computer can select the file from a pull-down menu and mount the encrypted volume.

They'll still have to get past your password though.
Once the encrypted volume is mounted you can save files to it just like you do with a normal volume.

TrueCrypt uses your modern hardware at its disposal to encrypt and decrypt files on the fly; which is to say it minimises the lag due to the overhead of converting unreadable bitstream into meaningful data that can be read by your text editor or played by your media player.

When you're through, unmount the volume with the Dismount button within the program.

Think formatting a disk is enough? Think again

Removing a file from the disk seems like a simple operation: just right-click on the file and send it to the trash.

Command line users may use the rm command do do the same thing.

Unfortunately, none of these methods actually deletes a file or a folder. They just hypnotise the filesystem to forget where a file is located in the disk.

These newly liberated disk locations are then added to the filesystem's pool of free address, and can point to new files.

That works in theory, but in practice the humongous size of partitions means that the disk locations that hold the deleted file may actually harbour them long enough for recovery tools to reconstruct them.



That's where shred comes in. Shred overwrites a file's space on the disk to make sure the space contains only garbage.

You might also want to use the --remove option to make sure it deletes the original file as well.

Shredding a file can be a lengthy affair, as it overwrites the location 25 times.

You can manipulate the number of rewrites with the -n switch, like this:

$ shred --remove -n 5 -v top-secret.txt
shred: top-secret.txt: pass 1/5 (random)...
shred: top-secret.txt: pass 2/5 (ffffff)...
shred: top-secret.txt: pass 3/5 (random)...
shred: top-secret.txt: pass 4/5 (000000)...
shred: top-secret.txt: pass 5/5 (random)...
shred: top-secret.txt: removing
shred: top-secret.txt: renamed to 0000
shred: 0000: renamed to 000
shred: 000: renamed to 00
shred: 00: renamed to 0
shred: top-secret.txt: removed

Shred works well on devices like /dev/sdb, which negates the use of the --remove switch, because you wouldn't want to remove the device.

There's a caveat here. Shred assumes the filesystem rewrites the file in place. This would render it useless on modern journalled filesystems such as ext3.

Shred also fails to wipe traces of the data being deleted in several places, such as the swap, RAM, and the filesystem journal.

An effective and secure deletion strategy requires the secure delete tools.


Secure-delete
The secure-delete tools include srm to securely remove the files, smem and sswap to wipe traces of data from the physical and SWAP memory, and sfill to ensure the free space on the disk doesn't point to old deleted files.

The tools make use of cryptographic algorithms especially designed to make sure deleted files are unrecoverable.

Once it's installed, make sure you remove the file or a directory with:

$ srm -v ../the-hole/eicar.com.txt

Using /dev/urandom for random input.

Wipe mode is secure (38 special passes)
Wiping ../the-hole/eicar.com.txt *********************************** *** Removed file ../the-hole/eicar.com.txt ... Done

Add the -r switch to recursively delete a directory. When you're done, make sure you wipe off residual traces from your RAM with smem, which may take a considerable amount of time depending on the size of the physical memory it has to wipe.

You can speed up the process with the -l switch, which reduces the number of rewrite passes (this is less secure).

Top off the process by disabling swap with swapoff , wiping it clean with sswap , and then re-enabling it with swapon.


The sfill command comes in handy when you are discarding a disk. Use it from a live CD on an unmounted partition to wipe the free space.


Remove junk
They might not be as bad as the other operating system, but all Linux distros tend to accumulate a lot of crud over a period of time. But why blame Linux?

The junk files are the legacy of the plethora of apps you have running on top of your kernel. You can pin their habit of collecting fluff to of the way the applications are configured to give you a better user experience.

And not only do all those log files, the temporary internet files and the various app caches accumulate to take up a considerable amount of disk space, they pose a great threat to your privacy.

Instead of trolling through the filesystem and emptying the various tmp/ directories, use BleachBit. It's a one-stop shop for removing all the crud that the apps have preserved.



BleachBit has a set of about 70 pre-defined cleaners, each of which works on a particular app such as Firefox, Google Chrome, Adobe Reader, OpenOffice.org and more.

The cleaners are tuned to wipe the dead weight off the applications and give them a performance boost.

The lightweight BleachBit is available in the repositories of all major distributions, though you might want to grab the latest build from its website. The project also releases bonus cleaner packs for older versions.

The BleachBit GUI is divided into two frames. On the left-hand side you select the apps that you wish to clean; this expands to give you more options specific to that app. In the right-hand frame, you get a brief explanation of each of these checkable options.


Get cleaning
To clean an area, such as Firefox's cache, simply click on the checkbox next to it. Some cleanup operations require you to trawl through a large location and involve more than a simple delete operation.

BleachBit will warn you when selecting such a task that might take up a considerable amount of time, for example, wiping the swap memory.

Before you ask BleachBit to zap the useless files in the apps you've selected, use the Preview button to review the list of files it'll delete.

If you encounter a file that you don't want to delete, such as the cache of a particular Firefox user, you can add it to a whitelist.

This is a list of files that BleachBit will not touch, even if the broader cleaner that they come under has marked them for removal.

You can specify any files or folders to bypass under the Whitelist tab under Edit > Preferences.

BleachBit also has a command line interface. For example, the following command cleans cookies under Firefox and Google Chrome:

$ bleachbit --delete firefox.cookies google_chrome.cookies

Use the --preview switch to get a list of files before removal. The CLI makes BleachBit scriptable for automated daily runs.

To add a cron job to nuke regularly created files, such as rotated logs and cookies daily at 2.00 am, edit the crontab with crontab -e and add the following line:

0 2 * * * bleachbit --delete firefox.cookies google_chrome. cookies system.rotated_logs

If daily sounds too frequent, you should at least run the app before creating backups. You can also use BleachBit to speed up certain apps, house clean the distro by fixing broken shortcuts, delete language packs and empty physical RAM and swap memory.

Browse anonymously
Pull a Keyser Soze on the internet – make it think you don't exist…
On the internet, sometimes the best form of privacy is being anonymous. It's difficult for an attacker to get to you if they can't pinpoint you on the network. And no one covers your tracks better than the combination of Privoxy and Tor.

Tor protects privacy via a distributed network of relays run by volunteers spread across the world. This helps prevent anybody monitoring your internet connections from learning what sites you visit.

Tor works with web browsers, instant messaging programs and many other TCP-based apps. But the various app protocols and associated programs can be coaxed into revealing information about the user, which is where Privoxy comes into the picture.

Tor depends on Privoxy and its filtering capabilities to enhance privacy.



Begin by pulling Privoxy from your distro repositories, then head into your browser's advanced settings where you can change its proxy settings.

Here just fill in 127.0.0.1 for the HTTP proxy, and specify 8118 as the port.

That's all there's to it.

When you're done, start the Privoxy daemon with /etc/ init.d/privoxy start. You can now access Privoxy's interface from http://config.privoxy.org or http://p.p.

To hook up Privoxy with Tor, you first need to set up Tor's package repository. This is easily done by adding the following line to your Ubuntu or Debian installation:

deb http://deb.torproject.org/torproject.org main

Replace with the name for your distro, like karmic, or sid. Then add the GPG key used to sign the packages by running the following:

gpg --keyserver keys.gnupg.net --recv 886DDD89 gpg --export A3C4F0F979CAA22CDBA8 F512EE8CBC9E886DDD89 | sudo apt-key add -

If you use Yum, create a torproject.repo under /etc/ yum/repos.d with the following content:
[torproject]
name=Tor and Vidalia
enabled=1
autorefresh=0
baseurl=http://deb.torproject.org/torproject.org/rpm/
DISTRIBUTION/
type=rpm-md gpgcheck=1
gpgkey=http://deb.torproject.org/torproject.org/rpm/RPMGPG- KEY-torproject.org

Again replace DISTRIBUTION with the name of your Fedora or CentOS release, such as centos5 or fc13.

Now fetch Tor via the package manager, which will also pull in additional packages like the Vidalia Tor GUI controller.

Make sure you don't install the Polipo web proxy app, since we are using Privoxy and the two might conflict because they operate on the same port.

The last step is to get Privoxy and Tor to talk to each other. For this just edit the Privoxy config file under /etc/privoxy and uncomment the following line:

# forward-socks4a / 127.0.0.1:9050

Also uncomment the following lines to make sure the local network is still reachable:
# forward 192.168.*.*/ .
# forward 10.*.*.*/ .
# forward 127.*.*.*/

Presto!

Now all our internet traffic that passes through the Tor and Privoxy proxies is masked.

Recording User Activity with a Script

Using Variables in Scripts
The purpose of this script is to log the commands and output of a user so you have an accurate record of all activity.  One problem that you find if you depend upon logs with users and sudo is that sudo will not log the stdout nor the stderr.

keystroke.sh
#!/bin/bash
# Capture keystrokes of a user and log
TIMESTAMP=$(date +%m%d%y%H%M%S)
HOST=$(hostname|cut -f1 -d.)
LOGDIR=/var/log/user
LOGFILE=${HOST}.${LOGNAME}.${TIMESTAMP}
touch $LOGDIR/$LOGFILE

# Set Prompt
export PS1=”[$LOGNAME:$HOST]@”‘$PWD> ‘
chown $LOGNAME ${LOGDIR}/${LOGFILE}
chmod 600 ${LOGDIR}/${LOGFILE}
script ${LOGDIR}/${LOGFILE}
chmod 400 ${LOGDIR}/${LOGFILE}


Analysis of the Script


#!/bin/bash
The bash shell is the shell to use with this script.


TIMESTAMP=$(date +%m%d%y%H%M%S)
This line creates a variable (the date followed by month,day,year,hour,minute,second).  Variables are symbolic names for memory in which you can assign values, as well as read the contents or manipulate the contents.

The advantage of a variable is that once it is assigned you can use it over and over.  When you create variables it is important not to place spaces around the “=” sign.

It is important to start and end your variables so the shell can tell where the variable ends, that is why you see examples of variable with ( ).

Note there must be a space after “date”.

Here is the name of the log, note the time stamp on the end.


m67.root.070909025935
HOST=$(hostname|cut -f1 -d.)
HOST is a variable that is created to indicate the machine logs that will be accessed.  It is created by two commands with the output of one piped into the second command.

The command hostname will print out the hostname of the computer the user is on.  That hostname could be a single hostname or it could be a Fully Qualified Domain Name (FQDN).

hostname
m67
or
hostname
m67.example.com


The hostname is piped into a second command with the “|” symbol which takes the output of one command and sends it to the second command.

So when you create the variable HOST the command is run and sent to the second command cut.  cut, as the name implies, is used to cut and display selected information from a text file or text input.

Think of it as something that will take a vertical slice of a text file, and send it to the output of your choice.

There are two ways to specify where you want to begin and end the slice.  You can specify it either by a starting and an ending character, or by fields.

To specify your “slice” by fields, you’ll need to use both the -d and -f switches.  The -d switch will specify the delimiter,  the character that separates the fields, in this case a dot.

That’s so that cut will know where each field begins and ends.   The -f switch will specify which fields you want to look at.

So the command you see with cut will take the first field and separate it from the other information that will be appended by a “.”.
m67.

If you wanted the see the first three fields of the hostname, FQDN, the script would be written like this:


HOST=$(hostname|cut -f1-3 -d.)
m67.example.com.


LOGDIR=/var/log/user
The variable $LOGDIR is created by determining the location of the log file after the “=”.  You can place the log wherever it is convenient.

LOGFILE=${HOST}.${LOGNAME}.${TIMESTAMP}
Here the $LOGFILE variable is created by using three previously created variables, separated by a “.”, note the brackets.


touch $LOGDIR/$LOGFILE
The command touch creates an empty file that can be used by the information that is recorded.  The “/” separates the two variables which have been determined by the text above in the script.

export PS1=”[$LOGNAME:$HOST]@”‘$PWD> ‘