PC Cleaning Apps are a Scam: Here’s Why and How to Speed Up Your PC

PC Cleaning Apps are a Scam: Here’s Why and How to Speed Up Your PC.

I get this question a lot, and I clean up a lot of computers after users use a cleanup tool.

Anyhow, this article has it explained simple and quick, and even shows what to do if you want to really speed up your PC.

Defragmentation is automatically scheduled in Windows Versions newer than Vista.

I want to remind you, that the best way to keep your PC running fast, is not to have unnecessary programs, making sure it is up to date, and running antivirus and antimalware often.

Computers will get a little slow over time, just because the Operative System (Windows/OSX/Linux) has updates and gets heavier, but also because our perception of what a fast computer is.

10 years ago web pages were 10 time smaller than they are now. So even web browsers are getting heavier.

Advertisements

Windows 8, Real impressions from an IT and gamer perspective

I have been using Windows 8 personally since it was released in TechNet.

Now, I want to clarify, that I used and liked Windows Vista, and then I loved Windows 7 when I moved to it a year later.. Because of my work, I try to use new Operative System as they are released and I try to have an open mind. Also because of my work I might not get to do it right away.

Started with Windows Vista

Vista got a lot of bad rep. I used it for over a year until 7 was released, and I never had a problem. It is more, Windows 7 is actually Windows Vista. The reason it got such a bad rep, was because it was installed on top (or side by side) with Windows XP, or in older hardware. Windows Vista required a good spec PC. Now, you could argue all you want about an Operative System that demands more resources, but the reality is, that I almost never consider installing a new OS in an older machine. Save your money, build or buy a better machine and then put the new OS, after all, think about the new features, new this, new that. In  the end the software will be bigger.

Only real problem Vista had, was the non use of 3D hardware for the 3D effects on the desktop.

OSX?

Now, before I go into Windows 8, think about OSX in the Apple Computers. Its current version is 10.8. Since the release of OSX, Apple has been making modification and releasing it under a new name, the latest ones being leopard (10.5), snow leopard (10.6), Lion (10.7), Mountain Lion (10.8). This is Apples biggest factor for growth and also the biggest reason for lack of innovation lately (you can barely tell apart a 10.8 install than from a 5 year older release Leopard (10.5). However users are comfortable with the user interface. Regardless though a lot of the OS actually changes, making it so some programs will not work in 10.7 from 105, or from 10.6 to 10.8. Technically for me the OSX ecosystem is a mess, but the number makes sense considering the graphical changes. Why I am explaining OSX? Because I don’t want anyone later saying “…Apple does this better…” or something like that. If you are happy with an Apple PC, good for you, but they are not perfect and Windows is neither, but we are talking only about software (OS) and not the hardware.

http://en.wikipedia.org/wiki/Osx For more information about OSX (Apple’s Operative System)

Now, onto the real meat. Windows 8

You might be wondering why I wrote about Windows Vista and OSX if I am writing about Windows 8? Windows 8 got a lot of bad rep before release because we as humans don’t take change well. We like to live on our comfort zone. To accept this change, I am listing very shortly how we got here.

Windows Vista was version 6.0 in the Code (Build 6002: Service Pack 2)

Windows 7 was version 6.1 in the code (Build 7601: Service Pack 1)

Ahh, you can you see where I am going with this?

Windows 8 is 6.2 (Build 9200)

In case you wonder about it, Windows XP was version 5.1.

Comparing to Apple’s naming, Vista would be Leopard, and Windows 7 would be Snow Leopard, while Windows 8 would be Lion. Microsoft and Apple have different release schedules and goals and that accounts why there are different numbers. It does not mean that Microsoft is behind. Windows XP was released 11 years ago in 2001. Vista was released in 2007 (just like Leopard).

Now think what changed in 5 years in Windows. 3 Versions but it went from build code 6002 to 9200. Microsoft took a lot more chances between releases. They could have released an interface that looks like XP and kept going at it. However it would get old soon. Don’t you think? Windows 7 definitively looks a lot more modern than Windows XP, and after working with 7 for 3 years I can happily say when I work in Windows XP I am less productive. Yes, less productive. And here comes the main point

If you learn to love Windows 8 you can be more productive

in 4 days Microsoft will raise the prices for Windows 8. If you have a fast machine that runs Windows 7 great, you have to give Windows 8 a try.

Start Screen. No more Start Menu

First things first. Get over it!! Plain and simple. The start menu is gone. OSX doesn’t have a start menu. It has a launcher and then the Applications folder. In Linux KDE has launcher, Ubuntu’s Unity has also a start screen launcher style. Same for Gnome3. In Linux people also complained about it when KDE switched to version 4, and Gnome from version 2 to 3.

The start screen/launcher makes sense. I will tell you about a problem I had in Windows XP.

I play games and I work in IT, that means my computers (at work and at home) have a lot of programs. My old home PC the start menu spawned 3 rows across the screen (1280×1024 resolution with small icons) And that is considering I tend to change the installation folder for games, and the program menu so they are all under “games”

With so many programs I spent too much time to open a program that I wanted. Regular users bypass this by using the desktop….

The Desktop is not a place to put files. It is a folder, with a special view, and it gets rendered as soon as the computer starts. If you had 200 icons on your desktop in Windows XP, you could see a real difference if you only had 2. The Start Menu only gets rendered when you go to it.

Windows Windows 7 I could start typing the name of a program and it gave me the possible results.

Windows 8 does the same.

The Start Screen is basically the Start Menu but full screen. What is great about it is that you can unpin applications and arrange the applications into groups. My start Screen is arranged in such a way that I can get quickly to anything I need or want.

What about for a regular user? If you clean your start screen then you will never had to scroll. All that you need is right there. As soon as the computer boot, you can go to your favorite application.

The Start screen is touch centric but works as well with keyboard and mouse. I don’t have a touch screen and I love it.

Enter the mobile era

Where the Start Screen shines is with mobile devices, like a tablet or phone. Ubuntu and KDE are also working on projects to have 1 interface for all devices. Think about the advantage. I have an Android phone that I needed to learn to use (it is not difficult though) I have a netbook that had Windows 7 and it was not a pleasant experience. Any I have my computer that I already know how to use and I use every day. If my phone, my tablet, my laptop and my computer all run Windows 8, then I would know how to use instantly all of my devices (with a few exceptions).

Windows RT

Windows RT is what powers tablets with Windows 8. RT is what runs the Start Screen. Any program (app) that can run in the Start Screen of a computer will run in a Windows RT tablet.

Lets be honest, tablets,phones and laptops are getting lighter and more powerful, but they are still not as powerful as the desktop counterparts. One main point is price. Windows Surface has 2 version, the RT which runs in a Nvidia Tegra 3 CPU (Used in tablets with Android 4 as well), and another which will be released in February which will use Intel Core i5. The Pro includes Windows 8 Pro, and can run regular Windows Applications, which basically makes it like a laptop.

The 2 models actually have a big price difference with the RT Version starting at 499$ (599 with keyboard cover), while the pro starts at $899. That is 400$ difference (or $300 if you compare to the 64GB RT Version).

You might be wondering what is the use if you cannot run your Windows applications on a RT tablet? Well, can you run your Windows application in your $399 iPad2? The answer is no.

But the applications you can run in your Windows 8 RT you can run in Windows 8 Pro. It is starting to make more sense.

Be completely fair while checking these products. The average user, and these are most users do generally the same range of activities over and over again, and it is generally consuming Internet content. Facebook, email, youtube, twitter, etc. You might have an app for that, but it in the end it is content that is online in a web page, and that is how the iPad did so well, it cater to those users. Windows RT aims for those users, but helps in the transition. A lot of applications even Sync, so you can run your app in your Surface RT, and then in your regular Windows 8 PC.

The end of the desktop era?

Lots of “technical” web sites have articles predicting the end of the desktop era. Microsoft acknowledges that it won’t happen with Windows 8, but that there is a tendency to separate work. I prefer to read Facebook in a tablet, so I can watch TV or be mobile. However, I will write documents, do network administration and other task that require more typing in a desktop. I could probably do most tasks that I do from a Windows RT tablet, but it would take me a lot longer than from my desktop (the same if I were to use an iPad).

So now that you know the goal. Does Windows RT make sense? Yes it does. And it is a trend that is here to stay, so we better get used to it. And if it makes it more productive, then better.

The Start Screen (RT) Applications (Apps)

Mail

The Mail app is very basic in look, with a high contrast. However one big improvement is that it lets you connect to an Exchange account. At this very moment I have my mail app open and I have my email from work (Exchange 2007), my personal web site (Office 365), Gmail and a Hotmail account. So effectively I don’t need Outlook. Now, the app is very basic, and does not support rules, but in the plus side it looks exactly like outlook.com, so if I go to Outlook.com to check my Hotmail email account, it is the same interface. I actually like the interface, but I do miss the rules. For a free app though it is just what I need to check my work email without having to open OWA (Outlook Web Access).

Music (xbox music)

This is my next favorite app. I suck at new music. I don’t use Pandora, iheart radio and other streaming. Why? because I already have a lot of music that I like. I put on MP3 all my CDs, so over the years I accumulated a lot of music. I have been using Google music because it allows me to put my music in the “cloud” and listen to it from my Android.

The Music app has free streaming in a lot of current music. For example, do you like Adele? You can listen to “21” and “19” for free.  What about Alicia Keys? Eminem, Linkin Park? I am listening now to “The Best of James Bond” as I write this. Every 3 or 4 songs there is a 15 minutes commercial. You can get a music pass and listen free of commercial

Do you like the album, then you can buy it through xbox music service. For example I really like the Skyfall version with Adele so I could purchase it since it is not available for free streaming.

Best of all, is that it integrates with my music in the music libraries. For example. I already have several albums from Linkin Park, so if I click on the featured for Linkin Park, the music that is local to my PC will play from here, no need to stream online and no commercial. So I can play Minutes to Midnight locally and Living Things streaming in the same playlist.

It is supposed to work with the Xbox too, but I haven’t tested it yet.

The only drawbacks I have found is that I cannot rate the music which I can do in Windows Media player. I use the rating a lot to create automatic playlist.

It also took me a while to realize that the search is part of the “charms” (use one of the right corner to bring the charms). I kept trying to just type as I do in the start screen.

It has been a while since I opened Windows Media Player so the music app is good overall.

Games

Have you tried the new minesweeper or solitaire? They look like paid apps now, but are free.

Now because I am a gamer I don’t play app games. They are exclusively for my phone for when I am sitting for a long time (at the doctors office, or other sitting :p),but I do like minesweeper and I spent several hours in one go playing daily challenges. It is a new take on a classic game. Or the Mine Sweeper Hero version.

Other Apps

There are more apps, but I don’t use them as often, and you might use different apps. But in general they work well. I haven’t had a single app crash yet, which is pretty good considering the platform is new.  Most of my Start Screen is shortcuts to regular desktop applications or video games.

The people app in an easy way to check updates from my contacts and not have to check the whole Facebook where you can spend hours just to catch up on 1 day, and I like the way it displays the “Whats new”

The Desktop

This is not RT, but regular Windows. Basically it looks the same as in Windows 7, but without the Start Menu button. I put my own, and I have not used it in months.

I keep my desktops pretty clean, with no icons on it. I like to see my wallpapers.

So for the desktop there isn’t much to comment. It is the same as Windows 7

Applications

Regular applications work as they did in Windows 7. Again nothing much changed.

So What are the changes beside the Start Screen?

Windows 8 is Windows 7 with Start Screen and RT apps. That is the main change, so it makes no sense why so many people think that Windows 8 is bad. I remember when Windows XP came out, people also said it was a bad OS but it ended being the most used Operative System. Windows 98 crashed a lot, and I had to re-install it a couple of times, so I was never a fan of 98.

Windows 8 has better support for Solid State Drives (SSD), and it boots faster. In the exactly same hardware Windows 8 is booting in 45 seconds (Core 2 Duo 8500) compared to 3 minutes that it took for Windows 7 (with a SSD boot drive). Thanks to this, I always turn off my computer instead of using sleep mode.

Windows 8 also includes better driver support. Windows 7 has excellent driver support, but I have a USB wireless card that I need to install the drivers manually but in Windows 8 is automatically detected and installed.

Future for Windows

I expect that Microsoft will keep improving Windows 8, as evident by the updates to the RT apps (They update through the store). With all the new ultrabooks, Windows 8 tablets and Windows phone I think that we will see a more consistent experience. As much as people like to complain about a new system, Windows is used on most computers. It makes total sense to me to get a companion device that works similar to my desktop but that it can use the device features (touchscreen for example).

It is evident that Microsoft will be more on schedule with versions and changes. Windows XP was 6 years old by the time Vista came out, but Windows 7 was released 30 months after. So we can expect Windows 9 in 3 years (or Windows Version 6.3), but it will be the same interface. Microsoft put a lot of effort into RT and it is a good gambit. I personally think it will be successful.

If there is one thing though that I want back is….MY SHUTDOWN ICON. It is easy to add, but it would have been better to be there from the start

In Short

Overall I like Windows 8, and I think it is an improvement over Windows 7. I will not buy any computer without it, or build a computer without it (unless it is a Linux computer), and I think you should seriously consider Windows 8 for your next computer

Windows 8 is not that different from Windows 7, so there is no reason to hold you back. The Start Screen is not a con, but it is a pro of the new Operative System. Unless you have an application that will seriously not work on Windows 8 (So far I only found one application, I already told the company that the lack of update for the application was a poor reflection on the commitment…This company is ADP BTW)

I cannot play Silent Hill 2, in Windows 7, and neither I can in Windows 8, but the game is from 2001, and since I can play it using Wine in my Linux PC I haven’t tried hard either.

On a side note. I love Windows Server 2012 🙂

Windows 8 Versus Windows 7: Game Performance, Benchmarked

http://www.tomshardware.com/reviews/windows-8-gaming-performance,3331.html

This is a good article for those not sure whether to switch to Windows 8. I have been using it for a while, and it does feel lighter at boot. I moved the desktop tile to the top, so I can get to the desktop with just a quick tap to the enter key.

In the summary it says “Aside from those couple of idiosyncrasies, performance under Windows 8 is indistinguishable from Windows 7. Any speed-up or slow-down would be almost impossible to identify during game play, and we expect compatibility issues to get patched quickly by game developers.”

So, it looks like there isn’t much of a problem switching to Windows 8. Currently the biggest problem is that we as humans don’t take change very well. However that does not affect gaming.

As a personal or business PC, I would still recommend to do the switch, however I would also recommend to invest time into learning the new “Windows Start Screen”, and the whole desktop in general. Even if you don’t have touch screen, the new start screen can be a big addition.

For example, I play video games, and I also use a ton of different programs for work. This makes usually my start menu quite long, and I have to scroll a lot to find the program that I want. With Windows 7 I started pinning to the start menu, and typing the name. Sometimes typing 4 letters of a program name is a lot faster than searching for it with the mouse (most of the time). With Windows 8, I was worried that my home screen was going to be super long. Well, it kind of was, but not for long. See, you can pin programs to the start screen, and you can also unpin them. That does not remove the shortcut from the Apps list, it just removes it from the Start Screen. So I removed everything that I don’t use often, I removed all Uninstall as well (I can still access them the “All Apps”), and I arranged my Start Screen putting the tiles I use the most in the first screen.

The end result is that my computer boots, it goes to the start screen and right from there I can click on the program that I need, be it MS Word, or a racing game.

I have some desktop icons, but honestly I am using them less and less, trying to leave my desktop uncluttered. Also, I have a pseudo Start Menu with all my Programs, but I haven’t used it in months, which means that the Start Screen is easier to use.

How long did it take me to organize my computer? Maybe 1 hour top.

Now, there is 1 thing I don’t like, which is the lack of a easy shutdown button, but there are some good articles online on how to do this

http://blog.laptopmag.com/how-to-shutdown-windows-8-in-just-one-click

http://www.howtogeek.com/77061/how-to-add-shutdown-restart-sleep-to-the-windows-8-metro-start-screen/

The value of Backup and Data

I have been bad….Very BAD!!!

When I started the blog, my second article was about home backups. I started writing in and before I knew it was too long. My problem was that I was covering too much ground, and too much information was fresh in my mind.

Well, after my article in NMS I decided to write this article, which will initiate a series of backup articles. Why a series? Because there is too much information that I need to give you, and I don’t want to bore you too much, but backup is so important and people give it so little importance that I MUST convey the information.

What is the Value of Data?

This is more than anything a question you need to ask yourself. The data that is important for me, might not be important for you. And Value is even harder to assign. However there is a some easy points to quantify it.

  • Can you reproduce the data?
  • Is it easy to host someplace else?
  • How vital is to have it available?
  • What is the damage if the data is loss (economical and emotional damage)
  • How often you need it?

None of these questions by themselves can put value on data, but all together can. For example, the perfect picture of the first birthday of your first child. You might not look at the picture all the time, but if lost forever, you will remember that picture and be sad about it (emotional value). However because it is a picture, it also probably is hosted in Picasa, Flicker, Facebook or another social media. Maybe not with the same quality, but at least you could save it from there if lost from your computer. In the other hand, lets say your QuickBooks database for your small/home business. Maybe it won’t slow you down a lot if lost, but could have several repercussions when tax season arrives.

Now that we have an idea how to assign value to data we should…

Assigning Value to Backup

The Value of the backup is proportional to the value of the data. It sounds like a lot of mathematical terms, but simply, if the data is very valuable (or invaluable) to you, then the backup is as well. VERY SIMPLE, No?

Although this is simple, I am still amazed to find out most people don’t backup regularly. I know I don’t backup most things at home almost never, however, most of the data I don’t care for (even though I have TB of data), and the data I care about I have it in my desktop, my laptop, my work PC, my old PC. It is a mess, but if needed I can recover most of it.

This mess of data also brings another value for the backup. The backup will organize your data. It is more is a consequence of backing up, but when you start to plan and put in effect a backup plan, you also end up organizing your data, which later will save you time when you are searching for it.

Extra benefits of backup

When setting a backup you will have plenty of benefits, some are more visible than others, however these are the ones I can think of right now:

  • Data can be recovered
  • Data will be replicated
  • Data will be organized
  • Data will be centralized
  • Multiple versions of the same data (*1)
  • When replacing computer, moving data is easier
  • Backup can be backup again (maybe to cloud, offline, remote)

(*1) It depends on the backup, but at least you have 1 older version of a file. Some backups support more than one version. If a file becomes corrupted, or you save a change that you want to undo later, an older version is the solution.

Thinking about value

Unless you do all your work online, and you don’t care about data, there is always something you won’t want to lose in your computer. Backing up is important, and have extra benefits beyond being able to recover data, and the value of the backup is the same as the data, or even greater in most cases.

So, do you value your data? And are you backing up your data?

Building the next generation file system for Windows: ReFS

Today I read this blog article from MSDN.

It talks about the File System ReFS in Windows 8. It is great to read at least the first part to understand what is coming in the new version of Windows. Most of these changes will be a big win for business where data needs to be more reliable and hot (always on), but are still great for home to keep our music, pictures and game saves Smile.

The article is very technical, getting more technical after  the section “Key design attributes and features” but at minimum is good to read the first part (5 minutes), and the Q&A at the end.

 

We wanted to continue our dialog about data storage by talking about the next generation file system being introduced in Windows 8.  Today, NTFS is the most widely used, advanced, and feature rich file system in broad use. But when you’re reimagining Windows, as we are for Windows 8, we don’t rest on past successes, and so with Windows 8 we are also introducing a newly engineered file system. ReFS, (which stands for Resilient File System), is built on the foundations of NTFS, so it maintains crucial compatibility while at the same time it has been architected and engineered for a new generation of storage technologies and scenarios. In Windows 8, ReFS will be introduced only as part of Windows Server 8, which is the same approach we have used for each and every file system introduction. Of course at the application level, ReFS stored data will be accessible from clients just as NTFS data would be. As you read this, let’s not forget that NTFS is by far the industry’s leading technology for file systems on PCs.

This detailed architectural post was authored by Surendra Verma, a development manager on our Storage and File System team, though, as with every feature, a lot of folks contributed. We have also used the FAQ approach again in this post.
–Steven

PS: Don’t forget to track us on @buildwindows8 where we were providing some updates from CES.


In this blog post I’d like to talk about a new file system for Windows. This file system, which we call ReFS, has been designed from the ground up to meet a broad set of customer requirements, both today’s and tomorrow’s, for all the different ways that Windows is deployed.

The key goals of ReFS are:

  • Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.
  • Verify and auto-correct data. Data can get corrupted due to a number of reasons and therefore must be verified and, when possible, corrected automatically. Metadata must not be written in place to avoid the possibility of “torn writes,” which we will talk about in more detail below.
  • Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.
  • Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.
  • Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.

The key features of ReFS are as follows (note that some of these features are provided in conjunction with Storage Spaces).

  • Metadata integrity with checksums
  • Integrity streams providing optional user data integrity
  • Allocate on write transactional model for robust disk updates (also known as copy on write)
  • Large volume, file and directory sizes
  • Storage pooling and virtualization makes file system creation and management easy
  • Data striping for performance (bandwidth can be managed) and redundancy for fault tolerance
  • Disk scrubbing for protection against latent disk errors
  • Resiliency to corruptions with "salvage" for maximum volume availability in all cases
  • Shared storage pools across machines for additional failure tolerance and load balancing

In addition, ReFS inherits the features and semantics from NTFS including BitLocker encryption, access-control lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks.

And of course, data stored on ReFS is accessible through the same file access APIs on clients that are used on any operating system that can access today’s NTFS volumes.

Key design attributes and features

Our design attributes are closely related to our goals. As we go through these attributes, keep in mind the history of producing file systems used by hundreds of millions of devices scaling from the smallest footprint machines to the largest data centers, from the smallest storage format to the largest multi-spindle format, from solid state storage to the largest drives and storage systems available. Yet at the same time, Windows file systems are accessed by the widest array of application and system software anywhere. ReFS takes that learning and builds on it. We didn’t start from scratch, but reimagined it where it made sense and built on the right parts of NTFS where that made sense. Above all, we are delivering this in a pragmatic manner consistent with the delivery of a major file system—something only Microsoft has done at this scale.

Code reuse and compatibility

When we look at the file system API, this is the area where compatibility is the most critical and technically, the most challenging. Rewriting the code that implements file system semantics would not lead to the right level of compatibility and the issues introduced would be highly dependent on application code, call timing, and hardware. Therefore in building ReFS, we reused the code responsible for implementing the Windows file system semantics. This code implements the file system interface (read, write, open, close, change notification, etc.), maintains in-memory file and volume state, enforces security, and maintains memory caching and synchronization for file data. This reuse ensures a high degree of compatibility with the features of NTFS that we’re carrying forward.

Underneath this reused portion, the NTFS version of the code-base uses a newly architected engine that implements on-disk structures such as the Master File Table (MFT) to represent files and directories. ReFS combines this reused code with a brand-new engine, where a significant portion of the innovation behind ReFS lies. Graphically, it looks like this:

NTFS.SYS = NTFS upper layer API/semantics engine / NTFS on-disk store engine; ReFS.SYS = Upper layer engine inherited from NTFS / New on-disk store engine

Reliable and scalable on-disk structures

On-disk structures and their manipulation are handled by the on-disk storage engine. This exposes a generic key-value interface, which the layer above leverages to implement files, directories, etc. For its own implementation, the storage engine uses B+ trees exclusively. In fact, we utilize B+ trees as the single common on-disk structure to represent all information on the disk. Trees can be embedded within other trees (a child tree’s root is stored within the row of a parent tree). On the disk, trees can be very large and multi-level or really compact with just a few keys and embedded in another structure. This ensures extreme scalability up and down for all aspects of the file system. Having a single structure significantly simplifies the system and reduces code. The new engine interface includes the notion of “tables” that are enumerable sets of key-value pairs. Most tables have a unique ID (called the object ID) by which they can be referenced. A special object table indexes all such tables in the system.

Now, let’s look at how the common file system abstractions are constructed using tables.

Object table: Object ID, Disk Offset & Checksum. Arrow to Directory: File name, File metadata; File Metadata: Key, Value; File extents: 0-07894, Disk offset and checksums; 7895-10000, Disk offset and checksums; 10001-57742, Disk offset and checksums; 57743-9002722, Disk offset and checksums

File structures

As shown in the diagram above, directories are represented as tables. Because we implement tables using B+ trees, directories can scale efficiently, becoming very large. Files are implemented as tables embedded within a row of the parent directory, itself a table (represented as File Metadata in the diagram above). The rows within the File Metadata table represent the various file attributes. The file data extent locations are represented by an embedded stream table, which is a table of offset mappings (and, optionally, checksums). This means that the files and directories can be very large without a performance impact, eclipsing the limitations found in NTFS.

As expected, other global structures within the file system such ACLs (Access Control Lists) are represented as tables rooted within the object table.

All disk space allocation is managed by a hierarchical allocator, which represents free space by tables of free space ranges. For scalability, there are three such tables – the large, medium and small allocators. These differ in the granularity of space they manage: for example, a medium allocator manages medium-sized chunks allocated from the large allocator. This makes disk allocation algorithms scale very well, and allows us the benefit of naturally collocating related metadata for better performance. The roots of these allocators as well as that of the object table are reachable from a well-known location on the disk. Some tables have allocators that are private to them, reducing contention and encouraging better allocation locality.

Apart from global system metadata tables, the entries in the object table refer to directories, since files are embedded within directories.

Robust disk update strategy

Updating the disk reliably and efficiently is one of the most important and challenging aspects of a file system design. We spent a lot of time evaluating various approaches. One of the approaches we considered and rejected was to implement a log structured file system. This approach is unsuitable for the type of general-purpose file system required by Windows. NTFS relies on a journal of transactions to ensure consistency on the disk. That approach updates metadata in-place on the disk and uses a journal on the side to keep track of changes that can be rolled back on errors and during recovery from a power loss. One of the benefits of this approach is that it maintains the metadata layout in place, which can be advantageous for read performance. The main disadvantages of a journaling system are that writes can get randomized and, more importantly, the act of updating the disk can corrupt previously written metadata if power is lost at the time of the write, a problem commonly known as torn write.

To maximize reliability and eliminate torn writes, we chose an allocate-on-write approach that never updates metadata in-place, but rather writes it to a different location in an atomic fashion. In some ways this borrows from a very old notion of “shadow paging” that is used to reliably update structures on the disk. Transactions are built on top of this allocate-on-write approach. Since the upper layer of ReFS is derived from NTFS, the new transaction model seamlessly leverages failure recovery logic already present, which has been tested and stabilized over many releases.

ReFS allocates metadata in a way that allows writes to be combined for related parts (for example, stream allocation, file attributes, file names, and directory pages) in fewer, larger I/Os, which is great for both spinning media and flash. At the same time a measure of read contiguity is maintained. The hierarchical allocation scheme is leveraged heavily here.

We perform significant testing where power is withdrawn from the system while the system is under extreme stress, and once the system is back up, all structures are examined for correctness. This testing is the ultimate measure of our success. We have achieved an unprecedented level of robustness in this test for Microsoft file systems. We believe this is industry-leading and fulfills our key design goals.

Resiliency to disk corruptions

As mentioned previously, one of our design goals was to detect and correct corruption. This not only ensures data integrity, but also improves system availability and online operation. Thus, all ReFS metadata is check-summed at the level of a B+ tree page, and the checksum is stored independently from the page itself. This allows us to detect all forms of disk corruption, including lost and misdirected writes and bit rot (degradation of data on the media). In addition, we have added an option where the contents of a file are check-summed as well. When this option, known as “integrity streams,” is enabled, ReFS always writes the file changes to a location different from the original one. This allocate-on-write technique ensures that pre-existing data is not lost due to the new write. The checksum update is done atomically with the data write, so that if power is lost during the write, we always have a consistently verifiable version of the file available whereby corruptions can be detected authoritatively.

We blogged about Storage Spaces a couple of weeks ago. We designed ReFS and Storage Spaces to complement each other, as two components of a complete storage system. We are making Storage Spaces available for NTFS (and client PCs) because there is great utility in that; the architectural layering supports this client-side approach while we adapt ReFS for usage on clients so that ultimately you’ll be able to use ReFS across both clients and servers.

In addition to improved performance, Storage Spaces protects data from partial and complete disk failures by maintaining copies on multiple disks. On read failures, Storage Spaces is able to read alternate copies, and on write failures (as well as complete media loss on read/write) it is able to reallocate data transparently. Many failures don’t involve media failure, but happen due to data corruptions, or lost and misdirected writes.

These are exactly the failures that ReFS can detect using checksums. Once ReFS detects such a failure, it interfaces with Storage Spaces to read all available copies of data and chooses the correct one based on checksum validation. It then tells Storage Spaces to fix the bad copies based on the good copies. All of this happens transparently from the point of view of the application. If ReFS is not running on top of a mirrored Storage Space, then it has no means to automatically repair the corruption. In that case it will simply log an event indicating that corruption was detected and fail the read if it is for file data. I’ll talk more about the impact of this on metadata later.

Checksums (64-bit) are always turned on for ReFS metadata, and assuming that the volume is hosted on a mirrored Storage Space, automatic correction is also always turned on. All integrity streams (see below) are protected in the same way. This creates an end-to-end high integrity solution for the customer, where relatively unreliable storage can be made highly reliable.

Integrity streams

Integrity streams protect file content against all forms of data corruption. Although this feature is valuable for many scenarios, it is not appropriate for some. For example, some applications prefer to manage their file storage carefully and rely on a particular file layout on the disk. Since integrity streams reallocate blocks every time file content is changed, the file layout is too unpredictable for these applications. Database systems are excellent examples of this. Such applications also typically maintain their own checksums of file content and are able to verify and correct data by direct interaction with Storage Spaces APIs.

For those cases where a particular file layout is required, we provide mechanisms and APIs to control this setting at various levels of granularity.

At the most basic level, integrity is an attribute of a file (FILE_ATTRIBUTE_INTEGRITY_STREAM). It is also an attribute of a directory. When present in a directory, it is inherited by all files and directories created inside the directory. For convenience, you can use the “format” command to specify this for the root directory of a volume at format time. Setting it on the root ensures that it propagates by default to every file and directory on the volume. For example:

D:\>format /fs:refs /q /i:enable <volume>

D:\>format /fs:refs /q /i:disable <volume>

By default, when the /i switch is not specified, the behavior that the system chooses depends on whether the volume resides on a mirrored space. On a mirrored space, integrity is enabled because we expect the benefits to significantly outweigh the costs. Applications can always override this programmatically for individual files.

Battling “bit rot”

As we described earlier, the combination of ReFS and Storage Spaces provides a high degree of data resiliency in the presence of disk corruptions and storage failures. A form of data loss that is harder to detect and deal with happens due to ­­­“bit rot,” where parts of the disk develop corruptions over time that go largely undetected since those parts are not read frequently. By the time they are read and detected, the alternate copies may have also been corrupted or lost due to other failures.

In order to deal with bit rot, we have added a system task that periodically scrubs all metadata and Integrity Stream data on a ReFS volume residing on a mirrored Storage Space. Scrubbing involves reading all the redundant copies and validating their correctness using the ReFS checksums. If checksums mismatch, bad copies are fixed using good ones.

The file attribute FILE_ATTRIBUTE_NO_SCRUB_DATA indicates that the scrubber should skip the file. This attribute is useful for those applications that maintain their own integrity information, when the application developer wants tighter control over when and how those files are scrubbed.

The Integrity.exe command line tool is a powerful way to manage the integrity and scrubbing policies.

When all else fails…continued volume availability

We expect many customers to use ReFS in conjunction with mirrored Storage Spaces, in which case corruptions will be automatically and transparently fixed. But there are cases, admittedly rare, when even a volume on a mirrored space can get corrupted – for example faulty system memory can corrupt data, which can then find its way to the disk and corrupt all redundant copies. In addition, some customers may not choose to use a mirrored storage space underneath ReFS.

For these cases where the volume gets corrupted, ReFS implements “salvage,” a feature that removes the corrupt data from the namespace on a live volume. The intention behind this feature is to ensure that non-repairable corruption does not adversely affect the availability of good data. If, for example, a single file in a directory were to become corrupt and could not be automatically repaired, ReFS will remove that file from the file system namespace while salvaging the rest of the volume. This operation can typically be completed in under a second.

Normally, the file system cannot open or delete a corrupt file, making it impossible for an administrator to respond. But because ReFS can still salvage the corrupt data, the administrator is able to recover that file from a backup or have the application re-create it without taking the file system offline. This key innovation ensures that we do not need to run an expensive offline disk checking and correcting tool, and allows for very large data volumes to be deployed without risking large offline periods due to corruption.

A clean fit into the Windows storage stack

We knew we had to design for maximum flexibility and compatibility. We designed ReFS to plug into the storage stack just like another file system, to maximize compatibility with the other layers around it. For example, it can seamlessly leverage BitLocker encryption, Access Control Lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks. We expect most file system filters to work seamlessly with ReFS with little or no modification. Our testing bore this out; for example, we were able to validate the functionality of the existing Forefront antivirus solution.

Some filters that depend on the NTFS physical format will need greater modification. We run an extensive compatibility program where we test our file systems with third-party antivirus, backup, and other such software. We are doing the same with ReFS and will work with our key partners to address any incompatibilities that we discover. This is something we have done before and is not unique to ReFS.

An aspect of flexibility worth noting is that although ReFS and Storage Spaces work well together, they are designed to run independently of each other. This provides maximum deployment flexibility for both components without unnecessarily limiting each other. Or said another way, there are reliability and performance tradeoffs that can be made in choosing a complete storage solution, including deploying ReFS with underlying storage from our partners.

With Storage Spaces, a storage pool can be shared by multiple machines and the virtual disks can seamlessly transition between them, providing additional resiliency to failures. Because of the way we have architected the system, ReFS can seamlessly take advantage of this.

Usage

We have tested ReFS using a sophisticated and vast set of tens of thousands of tests that have been developed over two decades for NTFS. These tests simulate and exceed the requirements of the deployments we expect in terms of stress on the system, failures such as power loss, scalability, and performance. Therefore, ReFS is ready to be deployment-tested in a managed environment. Being the first version of a major file system, we do suggest just a bit of caution. We do not characterize ReFS in Windows 8 as a “beta” feature. It will be a production-ready release when Windows 8 comes out of beta, with the caveat that nothing is more important than the reliability of data. So, unlike any other aspect of a system, this is one where a conservative approach to initial deployment and testing is mandatory.

With this in mind, we will implement ReFS in a staged evolution of the feature: first as a storage system for Windows Server, then as storage for clients, and then ultimately as a boot volume. This is the same approach we have used with new file systems in the past.

Initially, our primary test focus will be running ReFS as a file server. We expect customers to benefit from using it as a file server, especially on a mirrored Storage Space. We also plan to work with our storage partners to integrate it with their storage solutions.

Conclusion

Along with Storage Spaces, ReFS forms the foundation of storage on Windows for the next decade or more. We believe this significantly advances our state of the art for storage. Together, Storage Spaces and ReFS have been architected with headroom to innovate further, and we expect that we will see ReFS as the next massively deployed file system.

— Surendra

FAQ:

Q) Why is it named ReFS?

ReFS stands for Resilient File System. Although it is designed to be better in many dimensions, resiliency stands out as one of its most prominent features.

Q) What are the capacity limits of ReFS?

The table below shows the capacity limits of the on-disk format. Other concerns may determine some practical limits, such as the system configuration (for example, the amount of memory), limits set by various system components, as well as time taken to populate data sets, backup times, etc.

Attribute

Limit based on the on-disk format

Maximum size of a single file

2^64-1 bytes

Maximum size of a single volume

Format supports 2^78 bytes with 16KB cluster size (2^64 * 16 * 2^10). Windows stack addressing allows 2^64 bytes

Maximum number of files in a directory

2^64

Maximum number of directories in a volume

2^64

Maximum file name length

32K unicode characters

Maximum path length

32K

Maximum size of any storage pool

4 PB

Maximum number of storage pools in a system

No limit

Maximum number of spaces in a storage pool

No limit



Q) Can I convert data between NTFS and ReFS?

In Windows 8 there is no way to convert data in place. Data can be copied. This was an intentional design decision given the size of data sets that we see today and how impractical it would be to do this conversion in place, in addition to the likely change in architected approach before and after conversion.

Q) Can I boot from ReFS in Windows Server 8?

No, this is not implemented or supported.

Q) Can ReFS be used on removable media or drives?

No, this is not implemented or supported.

Q) What semantics or features of NTFS are no longer supported on ReFS?

The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas.

Q) What about parity spaces and ReFS?

ReFS is supported on the fault resiliency options provided by Storage Spaces. In Windows Server 8, automatic data correction is implemented for mirrored spaces only.

Q) Is clustering supported?

Failover clustering is supported, whereby individual volumes can failover across machines. In addition, shared storage pools in a cluster are supported.

Q) What about RAID? How do I use ReFS capabilities of striping, mirroring, or other forms of RAID? Does ReFS deliver the read performance needed for video, for example?

ReFS leverages the data redundancy capabilities of Storage Spaces, which include striped mirrors and parity. The read performance of ReFS is expected to be similar to that of NTFS, with which it shares a lot of the relevant code. It will be great at streaming data.

Q) How come ReFS does not have deduplication, second level caching between DRAM & storage, and writable snapshots?

ReFS does not itself offer deduplication. One side effect of its familiar, pluggable, file system architecture is that other deduplication products will be able to plug into ReFS the same way they do with NTFS.

ReFS does not explicitly implement a second-level cache, but customers can use third-party solutions for this.

ReFS and VSS work together to provide snapshots in a manner consistent with NTFS in Windows environments. For now, they don’t support writable snapshots or snapshots larger than 64TB.