30 Sep 2011
This is the final entry of a three part series on VMware Security Basics. Part 1 is here and part 2 is here. The first part describes a typical ESX environment and the minimum considerations for the security of the storage system, the second part describes security considerations for the ESX servers, and this part describes security considerations for the guest operating systems.
As a reminder, a typical ESX environment looks like the diagram below:
Guest Operating Systems
The thing to remember with guest operating systems is they are just like any other computer on the network and must be secured appropriately.
Secure the Guest OS
Each guest OS should be hardened to match the security context in which it resides and all security updates should be applied. If possible create a hardened template and use the template to deploy new VMs. The template should be updated regularly to apply new security updates. In addition to security updates for the OS, apply security updates for third-party applications as well.
Keep VMware Tools Updated
In most ESX environments, VMware Tools is installed on the guest operating systems. Just like any other software VMware Tools must be updated regularly to apply security updates.
Virtual Machines are Files
Virtual machines are files and can be copied and pasted like any other file. Control who is allowed to move or copy VM files and where VM files can be stored. Backups of virtual machines should be encrypted.
Backup Your Virtual Machines
Having a redundant storage system and replication does not take the place of backups. Virtual machine files can be corrupted like any other files. Replication will give you multiple copies of a corrupted VM, it won't allow you to restore a good copy of the VM.
Watch Out for Dormant VMs
Dormant VMs are machines that have been shutdown and left unused for a while. These machines are typically missing critical security updates and pose a risk to the network when they are brought back online. Virtual machines that are no longer needed should be backed up and removed from production ESX servers. If it is necessary to use a dormant VM, then it should be brought online in a test environment so that security updates can be applied before placing it back into production.
Conclusion
I hope you found this series of blog entries useful for securing your VMware environment. If you want more information please read the resources listed at the beginning of part 1, which is here.
13 Sep 2011
I received an e-mail from a recent college graduate asking me for career advice. I thought he asked some good questions and thought it would be a good idea to share his questions. I will share my answers as well but I would love to here answers from some of my readers.
As my degree is in business/information management, most of the coding that I know is self taught aside from 1 year in computer science. Is there really even such a thing as a security job that does not require alot of coding? I dont mind coding but there are other guys I have talked to that are head over heels for it. Do you think that would be a big pre-requisite for security?
There are many facets of information security and only a few of them require a lot of coding. If you want to do application security or exploit development then you need to do a lot of coding but other wise you can get by using shell scripting in Windows and Linux and by picking up another scripting language like ruby or python. I don't do much development, I usually use other people's scripts and modify them if necessary. I rarely write my own tools.
I also realise that security is not something that most people just step into right out of school. I also realise that people get into security from all kinds of paths and backgrounds, but what kind of jobs would you recommend I try and get to develop a solid foundation for what a security job requires?
The best way to get into information security is to learn the general principles and start applying them in whatever work you are doing. A lot of people start out as system administrators and move over to security from there. The key is determine the kind of work you love and then figure out how to apply information security to that work.
What is the worst thing about a pen testing/security job in your opinion?
For me the worst thing is seeing the same flaws over and over. It makes you feel like nobody cares or listens to you. I also don't like the way companies choose cheap security assessments over thorough security assessments. The security industry catered to the companies wants and we ended up with a bunch of charlatans doing sorry work. There are folks working to change this so maybe things will get better over the next few years.
Of the certifications that you have, which did you feel like you learned the most from? Which ones would you recommend for someone starting out?
I have a CISSP, GSEC, GPEN, and OSCP. Of those four the OSCP was by far the best certification. I am good at taking standardized tests so the other certs were relatively easy. The OSCP kicked my butt the first time I took it. I passed it on the second try but it took a lot of work. If you want to be a pentester the OSCP is the test to take.
What do you think about as far as employment for security jobs goes? Do you think there are enough companies/consulting/whatever firms that I would be able to eventually snag a job here if I was good enough? Or do you think moving to another city would probably give me the best opportunity? I know there is alot going on as far as security jobs go in the DC/Maryland/Virginia area.
The best advice I can give you is to find a city you love and an area of IT you love and start pursuing those two things. If you want to be a pentester then find a company that needs an intern and start learning what you can. If sysadmin type work is what you like then get a help desk job and move up from there. The trick is to find work you love and then figure out how security can be applied to the job.
Looking back at what you studied in college and on your own, what did you find the most interesting? What classes did you take that you thought were fun. What is your passion? After you determine your passion then you determine your path.
For more career advice visit Lenny Zeltser's blog. This article from Lenny Zeltser is the reason I started this blog and became more involved on LinkedIn and Twitter.
If you have any other advice for those starting out in an information security career then put it in the comments.
12 Sep 2011
This is part 2 of a three part series on VMware Security Basics. Part 1 is here. The first part describes a typical ESX environment and the minimum considerations for the security of the storage system. This part describes the minimum security considerations for the ESX servers.
As a reminder, a typical ESX environment looks like the diagram below:
ESX Servers
Four things to think about with the ESX servers are hardware redundancy, the server configuration, security update management, and the security levels of the guest operating systems.
Hardware Redundancy
Build each ESX server with redundant power supplies, processors, network cards, and host bus adapters (HBA). It is not enough to have a network card or HBA with two ports, two separate network cards and HBAs are necessary. In addition to redundant power supplies, consider using multiple UPS systems to provide battery backup to each server. The point is to ensure a single hardware failure does not take down the entire ESX server.
For a production environment run a minimum of two ESX servers to provide adequate fail over protection. If only two servers are running then each server should be capable of handling the maximum load. The better option is to run the number of ESX servers needed plus one additional server. This configuration allows for server maintenance without affecting the overall performance of the ESX environment.
Server Configuration
Harden each ESX server before placing it into production. The vSphere Hardening Guide gives a number of recommendations for hardening the ESX server configuration. The recommendations are based on three trust levels: enterprise, DMZ, and specialized security limited functionality. The enterprise recommendations are appropriate for most production environments and provide the minimum protections required by all major security and compliance standards, the DMZ recommendations are for environments that are particularly susceptible to targeted attacks, and the specialized security limited functionality recommendations are for specialized environments that are vulnerable to sophisticated attacks.
Security Update Management
Just like any other operating system the ESX server requires security updates, which should be installed on a regular basis. The simplest way to manage security updates is to use the built-in update manager in vCenter, which can scan each ESX server to produce a list of missing updates and then install the missing updates.
Security Levels of the Guest OS
Each ESX server should be dedicated to hosting guest operating systems (OS) of the same security level. A guest OS connected to the production network will likely be configured and maintained more securely than a guest OS connected to a test network. These two guests should not run on the same ESX server because a compromise on the less secure test machine could lead to a compromise on the more secure production machine.
Conclusion
As stated earlier, this is the second part of a three part series. I plan to conclude this series next week with security considerations for the guest operating systems. Until then, remember to build each ESX server with hardware redundancy and each ESX environment with server redundancy, maintain a secure configuration on the ESX server, install security updates on the ESX server, and keep guest operating systems of differing security levels on separate ESX servers.
05 Sep 2011
Virtualization is complex and there are many moving parts. I can not speak to all the details of hardening a VMware environment but I can speak to the minimum things to consider when installing or maintaining a VMware environment. For more advice, look at these documents:
A Typical ESX Environment
A typical ESX environment will have one or more ESX servers connected to a shared storage system such as a fiber channel or iSCSI SAN. Each ESX server will have one or more guest operating systems, each with VMware tools and a myriad of applications installed. This can be seen in the figure below:
In this environment there are three major areas of concern: the storage system, the ESX servers, and the guest operating systems.
Storage Systems
Four things to think about with storage systems are data availability, traffic isolation, the security levels of the ESX servers sharing the storage systems, and which ESX servers are allowed to see which data sets.
Data Availability
Whatever storage system is used, fiber channel or iSCSI, ensure there are multiple data paths between the storage system and the ESX servers. This includes dual controllers on the SAN, dual switches, redundant power sources for the SAN, and dual host bus adapters (HBA) on the ESX server. It is not enough to have a single HBA with dual ports, two HBAs are necessary. Before the system goes into production, testing should be done to ensure a single device failure does not prevent the ESX server from accessing the data.
Traffic Isolation
Traffic isolation is of particular concern in iSCSI systems because they use the same basic infrastructure as a standard network. All iSCSI traffic should be segmented from the rest of the network traffic to prevent an attacker from sniffing the iSCSI data. I am not a fan of using VLANs to segment traffic of differing security levels and always recommend physically segmenting iSCSI traffic from the rest of the network.
Shared Storage for ESX Servers with Differing Security Levels
ESX servers in differing security levels are configured and maintained differently. An ESX server setup as a lab environment is not going to be hardened to the same level as an ESX server holding the companies production systems and those two ESX servers should not share the same storage. An attacker who gained access to the weaker ESX server could use it to attempt to gain access to the production data on the shared storage system.
Share Data Volumes with the Appropriate ESX Servers
On a typical SAN, multiple data volumes are configured and each one is assigned a SCSI logical unit number (LUN), which is used to uniquely identify that volume. The SAN can then be configured to only allow specific HBAs to access specific LUNs. As an example, in a group of ESX servers only two of those servers may need access to the LUN that holds the HR data, the SAN should be configured so only the HBAs in those ESX servers have access to the LUN with HR data.
Conclusion
As stated earlier there are three major areas of concern with a production VMware environment, the storage system, the ESX servers, and the guest operating systems. I will discuss the latter two in upcoming blog entries. For now, remember to configure and test multiple paths to the data on the storage system, to isolate iSCSI traffic from the rest of the network, to keep ESX servers of differing security levels from sharing the same storage system, and to only share data sets with the appropriate ESX servers.
23 Aug 2011
I chose the handle averagesecurityguy because it is the best description of how I view myself. I know I'm not below average because I work hard at what I do and I've seen enough sorry work to know I'm not there. I know I'm not an expert because I've seen the work of experts in this industry and I know I'm not there. So I tend to see myself as average.
Over the years IT and infosec have often come into conflict with family obligations. Trying to keep systems up and running, trying to understand some new exploit, and trying to stay relevant in this ever changing field has caused more than one argument in my house. I have finally realized that this industry changes too fast and is too complex for me to waste my time becoming an expert. I have resigned myself to being average. I won't stop learning and I'm not giving up on infosec but I'm not killing myself for it either.
The beauty of this industry is that there are enough lazy sysadmins, poor programmers, and below average infosec practitioners to keep an average security guy employed for lifetime. Let me show you what I mean. I recently found this whitepaper over at the SANS reading room and was impressed by the timely and sage advice: physically protect your systems, minimize the number of installed packages and running services, use strong passwords, limit the number of admin users, update your systems, test your systems for vulnerabilities, monitor your systems periodically, and plan for disaster recovery. On a daily basis I see these simple rules violated, but the reason this paper makes me realize I will always have a job is that it was written ten years ago and at the time it was written this was not cutting edge advice.
In the last two months I have seen both the SNMP and SMB service available on Internet facing machines and I have compromised machines using MS08-067, which is three years old.
People keep worrying about advanced persistent threats (APT) but we still haven't figured out how to deal with basic daily threats (BDT). I'm not sure how to fix the industry but I do know that I can have a good career in infosec and never have to be above average.